Interactive Data View and Command System

ABSTRACT

A method and information system for capturing signals, for processing signals, and for providing signals at least partially based on, or bearing correlation to, the captured signals is disclosed. The information system includes a signal input unit, a wireless communication unit, and an output unit. The signal input unit (preferably an optical signal unit) is constructed and is positionable to capture signals associated with an eye. The output unit is constructed to provide information based on the captured signals or to provide information as a function of the captured signals or in correlation with the captured signals.

This application is a continuation of U.S. application Ser. No.11/783,408, filed Apr. 9, 2007, which is a continuation of U.S. Ser. No.10/066,292 filed Jan. 20, 2002, the entire disclosure of which isincorporated herein by reference, and claims priority of German patentapplication no. 10103922.0, filed Jan. 30, 2001.

FIELD OF THE INVENTION

The present invention relates to an interactive data view and commandsystem including a wearable signal input unit providing high level ofwearer comfort, performance and flexibility.

BACKGROUND OF THE INVENTION

Recording, processing and reproduction of images grows with anincreasing demand for information and their clear graphic visualization.Electronic image processing also involves processing of images taken bycameras, scanning systems and sensors in the visible light spectrum aswell as in other sections of the electromagnetic spectrum such as theinfrared, the radio frequency and the X-ray frequency range. Afterelectronic processing, many images are reproduced either as individualimages or as moving images on an image reproduction screen such as adisplay for presenting the information to the eye.

There are special image contents easier recognizable by electronic imageprocessing using, for example, local frequency filtering, marginsharpness increasing, image data compression, image correlation, dynamicreductions and false color-coding. On the other hand, other techniquesare concerned with the superposition or subtraction of auxiliary imagestaken from different spectral ranges or with the superimposing of storedplans, maps, and drawings onto the original image.

By applying image processing, the information content of the actual,direct image can be intentionally increased or reduced. Image processingis used in a wide range from increasing the image contrast toblending-in of additional information, marking of details, andhighlighting dangers. In many of these applications, it isdisadvantageous that the electronic camera is a “second eye system”separate from the human eye. This disadvantage is due to the fact thatthe images are seen from another recording location and thatadditionally, the pictures on the image screen are presented at anotherobservation location than the eye. Thus, the human eye must constantlychange between the direct observation and the indirect observation whiletaking into account different observation angles, different imagedetails, and different size ratios which leads to physical impairmentsand delays when decisions must be made.

These problems may be solved by the “head-up-display” (HUD) techniqueused in the piloting of combat aircraft, in that important informationsuch as instrument displays and target data are inserted or fade-in intothe open spectacles of the pilot's helmet and thus into the visual fieldof the pilot. This technique is also used experimentally in theautomobile industry for displaying of instrument readings on thewindshield so that the driver is not distracted from viewing the road byviewing the instrument panel.

The “virtual reality” or “cyber space” techniques use closed spectacles(i.e. glasses) where the outside view is blocked and three-dimensionalfull images are projected into the eye with virtual reality. Thesevirtual reality-images are then modified in response to body motionssuch as locomotion, movement of an arm,

a finger, or head and eye movements.

There are other HUD techniques and image detection and projectiontechniques described in the PCT Application PCT/EP97/04188 (published asWO98/05992) and U.S. Pat. No. 6,227,667. The apparatus described inthese documents capture (or detect) the retinal reflex image and alsoenable a superimposition of supplementary images in the eye.

There is still a need for an information system capable of providinginteractive data view and command applications for use in variousenvironments.

SUMMARY OF THE INVENTION

The present invention is a method and information system for capturingsignals, for processing signals, and for providing signals at leastpartially based on, or bearing correlation to, the captured signals. Theinformation system includes a signal input unit, a wirelesscommunication unit, and an output unit. The signal input unit(preferably an optical signal unit) is constructed and is positionableto capture signals associated with an eye. The output unit isconstructed to provide information based on the captured signals or toprovide information as a function of the captured signals or incorrelation with the captured signals.

The signal input unit may be a non-optical signal unit for capturingnon-optical signals (e.g., acoustic data, electric data, magnetic data),or an optical signal unit for capturing optical data (e.g., ultraviolet,visible or infra-red). The optical signal unit can capture data byscanning or large image detection using light sensitive arrays likeCCDs. In some embodiments, the signal input unit may be integrated withthe output unit, and may also include an information unit.

Preferably, the optical signal unit and possibly other units areconstructed in the form on spectacles or goggles worn by a user, asdescribed in the PCT Application PCT/EP97/04188 (published asWO98/05992) and U.S. Pat. No. 6,227,667; U.S. application Ser. No.09/462,440; or PCT Applications PCT/EP00/09840, PCT/EP00/09841 andPCT/EP00/09843, filed on Nov. 6, 2000; the content of these applicationsis explicitly incorporated by reference into this application as iffully provided herein. The various embodiments and units described inthe above patent documents have been further improved to createeconomical, user-friendly, comfortable, high performance devices. Inparticular, adjustments and improvements of the aforementioned opticalunits provide a novel manner of projecting images onto or in front ofthe user's eye, including the retina and other surfaces. The presentoptical signal unit and output unit are not limited to scan-like captureof the retinal reflex image or projection on the retina, but utilizeother properties and function of the eye and human perception.

Since the apparatuses and systems described in the aforementionedapplications are preferably embodied in the shape of spectacles, theywill also be designated hereinafter, for the sake of simplicity, as aspectacle system, but this designation does not and should not imply anylimitation. The optical apparatus that scans the retina and displayscorresponding image modifying or image improving optical informationonto the retina are optimally used.

The signal input unit may be an optical signal unit is constructed tocapture visible signals, infrared signals or ultraviolet signals. Theoutput unit may be constructed to provide the information on a displayby projection or using a graphical user interface. The output unit maybe constructed to project the information. For example, the output unitis constructed to project at least part of the information onto theretina of the eye, or the output unit is constructed to project at leastpart of the information onto a two or three dimensional surface remotelylocated from the eye. The output unit may be constructed to provide theinformation in an audible format. The output unit is constructed toprovide the information in a format causing sensory skin stimulation,stimulating tactile organs, or visual, audible, smellable, or tastablesenses.

According to another aspect, an information system includes an opticalsignal unit, a wireless communication unit, and an output unit. Theoptical signal unit is constructed and positioned to capture signalsassociated with an eye. The output unit is interfaced with the wirelesscommunication unit and is constructed and arranged to provideinformation using a correlation unit. The correlation unit isconstructed to find suitable relationship between the captured signalsand additional data.

Preferred embodiments of this aspect include one or more of thefollowing features:

The information system may also include an information unit. Theinformation unit may include various databases, processors, sensors, aninformation network connection, an evaluation unit, and other devicesproviding additional data to the information system.

The correlation unit is constructed to determine a presentationrelationship of the captured signals and the additional data. Thecorrelation unit is constructed to determine the presentationrelationship between the captured signals and the additional data interms of location of the additional data with respect to the captureddata. Alternatively, the correlation unit is constructed to determinethe presentation relationship between the captured signals and theadditional data in terms of presentation timing of the additional datawith respect to the captured data.

The correlation unit may be constructed to determine the presentationrelationship between the captured signals and the additional data interms of relative color display of the additional data with respect tothe captured data. Alternatively, the correlation unit is constructed todetermine the presentation relationship between the captured signals andthe additional data in terms of relative size of the additional datawith respect to the captured data.

In short, the correlation unit performs the correlation function thatmay be “one-to-one” overlap of a scanned and a projected image, or maycompletely differ from the “one-to-one” overlap, or may includedifferent type of data. For example, the input signal unit may include amicrophone to the external system, in particular a directionalmicrophone, which is directed as a function of the head position or thegaze direction to allows a further sensory dimension to be realized.

The correlation unit may direct projection of a correlated field-of-viewimage of an individual (e.g., a fireman) or a mosaic of several imagesinto the eye or on a projection surface of another individual (e.g., afire chief overseeing the operation). The correlation unit enables, forexample, the projected image of the entire scene (field-of-vision) to bepartially overlapped by data symbolizing the position and severity ofthe sighted or otherwise captured (or detected) event (e.g., firelocation and temperature colored in accordance with the strength of thefire, which could be blended into each fireman's view). Alternatively, adisruptive effect of the natural field of view not necessarily coveredup by the projection of information can be reduced via acomplementary-color “wiping out” in which complementary-colored pixelsare determined on the basis of the light captured from the field of viewwhose correlated projection onto the respectively associated regions ofthe retina make the natural background appear white on account of theaddition of colors. The virtual position of the fixation fixed, e.g.,via a fixation with the eye in conjunction with a blinking of the eyes(or a pressing of a button or even automatically), for example, by animage-processing evaluation of the field of vision that determines anarea of the field of vision having as little content as possible.

According to yet another aspect, an information system includes anoptical signal unit, an information unit, a wireless communication unit,and an output unit. The optical signal unit is constructed andpositioned to capture signals reflected back from at least one eyecomprising the retina. The field-of-view capturing unit is constructedand arranged to capture light from a field of view associated with theretina without capturing a retinal reflex image. The output unit isconstructed to provide information, at least partially obtained via thecommunication unit, in cooperation with the information unit as afunction of the captured light and in correlation with the capturedsignals.

According to another aspect, an information system includes an opticalsignal unit, an information unit, a wireless communication unit, and anoutput unit. The optical signal unit is constructed and positioned tocapture signals reflected back from at least one eye comprising theretina. The optical signal unit comprises a scanning detection unitconstructed to at least partially capture a retinal reflex image of theretina. The output unit is constructed and arranged to provideinformation, at least partially obtained via the communication unit, incooperation with the information unit as a function of the capturedsignals.

The output unit may be cooperatively constructed and arranged with theoptical signal unit to enable projection of information onto the retina.Alternatively, the output unit is not constructed to project informationonto the retina.

According to yet another aspect, an information system includes anoptical signal unit, an information unit, a wireless communication unit,and an output unit. The optical signal unit is constructed andpositioned to capture signals reflected back from at least one eyecomprising the retina. The optical signal unit comprises a scanningdetection unit constructed to at least partially capture a retinalreflex image of the retina during a first scanning operation andcarrying out a less comprehensive capture of the retinal reflex imageduring a later scanning operation. The output unit is constructed andarranged to provide information, at least partially obtained via thecommunication unit, in cooperation with the information unit as afunction of the captured signals. The output unit comprises a scanningprojection device constructed to project at least part of theinformation onto the retina.

According to yet another aspect, an information system includes a signalinput unit, an information unit, a wireless communication unit, and anoutput unit. The signal input unit is constructed not to capture anysignals reflected back from the retina. The output unit is constructedand arranged to provide information, at least partially obtained via thecommunication unit in cooperation with the information unit as afunction of the captured signals. The output unit comprises a scanningprojection device constructed and arranged to project at least part ofthe information onto the retina.

According to this aspect, the signal input unit may be an optical signalunit constructed and positioned to capture signals reflected back fromat least one eye but not from the retina.

According to yet another aspect, an information system includes a signalinput unit, an information unit, a wireless communication unit, and anoutput unit. The signal input unit is constructed and positioned tocapture at least two types of signals wherein one or both signals may bereflected back from the eye. The output unit constructed and arranged toprovide information, at least partially obtained and/or provided via thecommunication unit, in cooperation with the information apparatus as afunction of the captured signals. The output unit comprises a scanningprojection device constructed to project at least part of theinformation onto the retina of the eye.

Preferred embodiments of the above aspects include one or more of thefollowing features:

The information system includes a spherical or spherical-actingreflection layer operably positionable at a location immediatelyanterior and substantially confocal to the eye, and the optical signalunit is constructed to capture optical field-of-vision signals reflectedoff the spherical or spherical-acting reflection layer.

The field-of-view capturing unit is constructed to capture visible lightfrom a field of view associated with the retina without capturing aretinal reflex image thereof. The output unit is suitable for providingthe information in correlation with the captured visible light.

The information unit comprises an evaluation module constructed toobtain image information with regard to the field of view from thecaptured visible light.

The output unit comprises a projection device constructed to project theimage information onto the retina in correlation with the capturedsignals such that a naturally perceived field of view and projectedimage information are perceived as a unitary image by the retina. Thescanning and projection may utilize separate beam paths.

The information system provides a function that encompasses a temporalor spatial correlation between the provision of information and thecaptured visible light. This function may also encompasses patternrecognition that yields at least one information key, and theinformation keys serve for an information query based on the informationapparatus.

The optical signal unit includes a scanning device that records an atleast partial capture of a retinal reflex image of the retina in a firstscan operation and carries out a less comprehensive capture of theretinal reflex image in a later scan operation. The optical signal unitmay capture a retinal reflex image of the retina only partially or notat all.

The field-of-view capturing apparatus includes a spherical orspherical-acting reflection layer suitable for deflecting a portion of alight directed at the eye into a sensor apparatus for capture. Thefield-of-view capturing unit or the optical signal unit may be suitablefor at least partially capturing a corneal reflex image of the eye. Theoptical signal unit or the field-of-view capturing unit may befabricated as portable units. The information unit may include adatabank (including various storage devices), a sensor system, aninformation network connection and/or an evaluation unit. Theinformation unit may be fabricated as a portable unit.

In accordance with one aspect of the optical input unit, the opticalapparatus is wearable by an operator and capable of scanning the retinaup to frequencies of 100 Hz and an image falling onto the eye, inparticular a retinal reflex image. The optical input unit is a part ofan interactive data view and command system, with the particularadvantage that, for operating the system, a least possible effort on thepart of the operator is necessary. The information to be queried can betransmitted to the operator with shortest possible delay either in theform of image signals and/or in the form of signals for controllingfurther information reproduction devices that can be operated e.g. on anacoustic or other sensory basis. A system thus results that isdistinguished by a highest measure of directness of the interactiveinformation exchange.

The data transmission unit may include a mobile communication system isswitched between the data view and command system and an externalinformation source. In this manner, the field of application of theentire information system is additionally extended. The datatransmission unit can include a mobile telephone or a computer, e.g., alaptop or palmtop having a suitable data remote transmission interface.

The system can execute additional control commands that control the flowof information. These control commands can be based on opticalinformation signals that are output by the optical apparatus. Forexample, a menu bar is blended in to the operator into the field of viewby outputting image signals onto the retina via the optical apparatus,again with an image frequency of roughly up to 100 Hz. The selection ofa menu item is carried out either via a control mouse, or instead solelyvia the focusing of this selected menu item and with the aid of aselection signal. The selection signal can be a button, voice command oran eye blink signal. The focusing can be determined optically since theselected menu item is situated in the center of the fovea centralis,which can be determined, in parallel, while the scan of the retina/ofthe image falling onto the eye is running.

The signal input unit may include a microphone and a speech analysisunit.

The operator's additional control commands can be based on acousticinformation signals that are output from a speech input unit. Since, thesignal input unit may include, an optical apparatus is used thatcyclically scans the operator's retina, and the system can be employedto carry out an active operator recognition in a highly economicalfashion. It is thus possible, on the one hand, to automatically protectthe system/system operations from unauthorized use and, on the otherhand, to carry out an automatic personalized set-up of the system to theoperator wearing the system. For this purpose, it is solely necessary totemporarily store the retina structure captured by the optical apparatusin the form of a data record in the signal processing apparatus and tocompare it to a person-specific data record already stored.

Due to the compactness and the short signal paths from the opticalapparatus to the eye, on the one hand, and to the communicationapparatus on the other hand, application of the system is quiteversatile including the use in the realm of medicine, in particular inthe field of ophthalmology, as both a therapeutic and analytic device.

Additional applications are described in the PCT ApplicationsPCT/EP00/09840, PCT/EP00/09841 and PCT/EP00/09843, which areincorporated by reference.

The present information system includes numerous embodiments thatprovide one or more of the following features:

at least partially captures a cornea reflex image of the eye;

directs a part of the light falling onto the eye into a sensor apparatusby using the spherical or spherical-acting reflection layer;

determines the retinal image via the degree of oxidation of the retinalcones and/or the retinal rods;

carries out solely a partial capture of a retinal reflex image; and/or

comprises a field-of-view capturing apparatus that captures visiblelight from the naturally perceived field of view without capturing aretinal reflex image.

At the same time, the present system in accordance with the describedembodiments can provide:

information as a function of a naturally perceived field of view of aperson;

information as a function of signals captured from an eye, yet does notproject these into the eye from which the signals were captured; orinformation as a function of signals captured from an eye, wherein theinformation is at least partially projected into the eye, the signals,however, are not captured in the manner known from the PCT ApplicationPCT/EP97/04188 (published as WO98/05992) or U.S. Pat. No. 6,227,667.

The signal input and output units, including the described spectaclesystems, are based on a combination of several of the aforementionedfundamental concepts, the result of which is a natural interrelationshipof the associated three applications. The present information systemsallow additional information to be provided to the operator that goesbeyond our personal knowledge and sensory impressions. Examples of thisinclude searching for an electrical cable under the plaster in a wall,navigating in an unfamiliar city, collecting wild mushrooms, andinspecting a possibly dangerous object using a remote-controlled robot.The present information system can provide information to a person oranother system.

The strong dependency of a person having vision on their visual sensesclearly contributes to the difficulty of providing additionalinformation. Indeed, the fact that people having vision primarily usetheir eyes makes it necessary, in many cases, to either input thesupplementary information via the eyes or to determine the supplementaryinformation based on the information seen. In the case of input via theeyes, however, the orientation of the eyes must be taken into exactconsideration for correct “placement” of the input information and toavoid a “jittering” or “blurring” thereof. In addition, in many casesthe information should be made available without a controlled movementof the eyeballs; a car driver may have a map on his lap but would prefernot to have to look away from the street.

Due to their dependency on solid media, e.g. paper, CRT and LCD screens,etc., prior visual information systems have not been in a position tosufficiently fulfill the comfort needs of a person having vision.Non-visual information systems previously lacked the correlation to thatwhich is seen that is natural for people having vision.

Through the incorporation of the interactive data view and commandsystem into an information system in accordance with the dependentclaims, a system is provided whose presentation of information fulfillsthe natural needs of person having vision in a previously unachievedmanner. At the same time, the information system is improved over theprior art with regard to its implementability and economy.

In its most general form, the information system in accordance with theinvention comprises a signal capturing apparatus that captures signalsreflected back from an eye comprising a retina, an information apparatusand an output apparatus that provides, in cooperation with theinformation apparatus, information in correlation with the capturedsignals. Preferably, the information is provided as a function of thecaptured signals and/or as a function of visible light captured from thenaturally perceived field of view.

Preferably, one of the aforementioned spectacles in which a scanningdetection apparatus at least partially captures a retinal reflex imageof the retina serves as a signal capturing apparatus. A modification ofthis detection apparatus that captures light reflected on a cornea ofthe eye in lieu of the retinal reflex image is particularly advantageousfor infrared applications since the cornea strongly reflects lighthaving a wavelength of roughly 1.1 μm. It is also fundamentally possibleto make correspondingly valuable statements about the image falling ontothe retina by capturing the chemical change of the rods and/or cones.

The present system uses the advantages of capturing the field of viewcomplementary to the capturing of signals reflected back from the eye.For the purpose of such a complementary capturing, the field-of-viewcapturing apparatus and/or the information apparatus of the informationsystem in accordance with the invention preferably comprises a sphericalor spherical-acting reflection layer that is positioned essentiallyconfocal to the eye that deflects a part of the light directed onto theeye into a sensor apparatus for capture. Due to the fact that thereflectivity of the reflective layer is several times higher than thatof the retinal or corneal reflex, considerably more light can becaptured using equally sensitive photo-sensors. Moreover,correspondingly cheap photo-sensors could be used in the sensorapparatus. It can also be advantageous if the light falling onto the eyeis not solely, only partially or not at all captured via the retinalreflex.

Depending on the intended application, not all spatial regions of thefield of view must be captured. In an application, for example, in whichsupplementary information with regard to an object upon which the eyehas fixed its gaze is provided by an information system in accordancewith the invention, it could be sufficient to capture the light fallingonto the fovea and to subject it to a pattern detection or other type ofanalysis since an object upon which the eye has fixed its gaze istypically imaged on the fovea which represents the area of keenestsight. Accordingly, capturing the light that falls onto this part of theretina would possibly be sufficient to determine a sufficient number ofcharacterizing object features.

It is also sensible if only a limited spectral range of the lightfalling onto the eye is detected. If, for example, the infrared lightfalling onto an eye is detected, the orientation of the eye and/orvaluable information from the field of view can be determined, even atnight.

Accordingly, any limitations with regard to the capturing of the lightfalling onto an eye can be meaningful. In particular, limitations of thecaptured spectral range, the captured regions of the field of viewand/or the captured time spans of vision are applied as necessary.

For the purpose of redundant or stereoscopic image capture, thecorrespond apparatus of the information system in accordance with theinvention can be designed so as to capture the light falling ontoseveral eyes. Depending on the field of application, the eyes must notnecessarily belong to a single person. For example, it would be possibleto display the images perceived by the eyes of several firemen ontomonitors in a command center in addition to position and fire strengthinformation determined from an infrared spectral analysis of the images.

In the field of ophthalmology, a distinction is made between the terms“field of view” and “field of vision.” A field of view is the part of aspace that can be seen with a stationary eye. A field of vision is theregion that can be seen with the eyes. Consequently, here, as elsewhere,the field of view is to be understood as the cause of the light thatnaturally falls onto an eye.

The present system may use a field-of-view capturing unit (i.e.,field-of-view capturing apparatus) for capturing light from the field ofview associated with the eye of a particular quality, i.e. with asensitivity, a resolution, a sharpness, etc. This field-of-viewcapturing of visible, infrared or ultraviolet light far exceeds thenatural visual acuity of the eye. The field-of-view capturing unit isnot limited to capturing of the field of view, but may include partialor complete capturing of the field of vision that encompasses an atleast partial capturing of the field of view.

The field-of-view or field-of-vision capturing units capture highquality images that can also serve as a basis for an extrasensorypresentation of information. For example, field-of-view information canbe obtained from the captured light of the field of view and projectedonto the retina such that the image seen by the eye seems at leastpartially sharper, closer, wider angled or in some other mannerextrasensory.

The present system enables novel ways of correlating and presenting dataacquired, computed (manipulated), received via a wireless communicationunit, or from a storage device. Importantly, the provision ofinformation with the signals captured by the signal capturing unit, thesystem treats the corresponding parts of the captured light during aprocessing that occurs in the course of the provision of information asif they were reflex images captured from the eye, i.e., as if they werethat which is truly seen. The information system can also combine highquality field-of-view information directly from the truly seenfield-of-view information from the eye.

The correlation of the presentation of information with the capturedsignals reflected back from the eye can be carried out, for example, bycapturing several pixels of an ocular reflex image, e.g. a cornea orretinal reflex image, that are brought into connection withcorresponding pixels from the captured field of view via an evaluationapparatus. A gaze direction of the eye determined via the capturedsignals can also serve to establish a correlation between the field ofview information obtained from the captured field of view and which istruly seen. As will be described herein below, the correlation can,however, also comprise projecting obtained field of view informationonto the retina in a correlated manner to that which is seen.

The information unit may include a data bank, a sensory unit, aninformation network connection, one or several processors, and/or anevaluation unit. The term “evaluation unit or apparatus” means any typeof evaluation apparatus and in particular image processing devices. Theinformation unit can include one or several sensory units collectingdata tactually, visually, audibly, smellably and/or tastably.

The sensory unit provides an extrasensory perception in connection withthe visible data. For example, when searching for an electric cable, thesensory unit can include one or several magnetic field sensors capableof localizing metallic cables with regard to a known coordinate system,e.g., the captured field of view. Image processing software superimposessensor signals from existing electrical cables and provides asupplementary image, as described in aforementioned patent applications,projected onto the image seen by the eye.

The information unit may also include other types of sensors used as aninformation source, in particular when the sensor is activated and/orqueried on the basis of the captured light image. For example, duringinspection of an integrated electronic circuit the position of aconductor on a manufactured chip could be computed after a directed gazeat that conductor on a circuit plan of the circuit and the pressing of abutton so that the current and voltage values of the conductor aredetermined using the non-contacting measuring device and presented tothe use via the spectacle unit.

The information unit may also include a data bank and an informationnetwork connection. For example, is an intra-company mail distributionsystem files include bar code stickers that uniquely identify therespective file. If a file is to be sent within the company, the senderenters e.g. the receiver's extension and a code designating the fileusing software that correspondingly stores these data in a data bank inone of the many known ways. When the file is later sorted, theidentifying bar code is captured via the spectacle unit worn by a maildistribution employee, e.g., via a directed gaze and a push of a button,and recognized via a recognition apparatus or recognition software. Thedata associated with the file relevant to mail distribution areretrieved from the data bank via a radio connection to an intra-companydata network and these data are presented to the mail distributionemployee, after pre-processing, if necessary, via a suitable outputapparatus, e.g. as an announcement via headphones “Mr. Schmidt, financedepartment, building G, second floor, room 310.”

The presentation of information that complies with the needs of a personhaving vision in a manner not previously achieved. This can compriseproviding the information to the person in a suitable manner, i.e. usingone or more of the five senses. The information can, however, bepresented in an arbitrary manner and does not require a particularaddressee. For example, the information can be provided to a furthersystem or radiated into the environment via an optical or acousticoutput apparatus. The claimed dependency between the provision ofinformation and the light image falling onto the eye guarantees that thecorrelation that a person having vision expects exists between theinformation provided and that which is seen.

This dependency is taken into consideration during the determination ofthe information, during the provision of the information or during bothof these inherent processes. Examples for establishing this dependencyduring the determination of the information are given above. During theprovision of information, this dependency can be established, forexample, by blending the information into the image seen by projectingback into the eye in such a way that a temporal, spectral, spatial,contractual or other sensible correlation is established between theinformation and the image seen. In particular, the dependency canconsist of the captured light image being used to determine the positionand orientation of the eyeball so that an image projected onto the eyefor the sake of providing the information appears to stand still duringa motion of the eye, appears to move with the eye during a motion of theeye or appears to move in accordance with a predetermined course duringa motion of the eye. In particular, the effect of the saccadic motion ofthe eye on these processes can be taken into consideration and/orcompensated.

The optical signal unit or the output unit (or the spectacle apparatus)can determine the position and orientation of at least one eye quickly,accurately and at little expense, e.g. at a determining rate of 100 Hz,a positional accuracy of several micrometers and using a portablyconstructed apparatus. By using the information during the dynamicevaluation of the orientation of the eye, the processing can be carriedout so quickly that the accuracy is not impaired by the saccadic motionof the eye. This is achieved by a signal-capturing unit, which does notcontact the eye and captures signals reflected back from the eye.Reflectable signals, e.g., sound or electromagnetic signals, allow ahigh frequency capturing such that the processing speed is primarilydetermined by an evaluation unit. The evaluation unit may have,depending on the application, very high processing speed, low powerconsumption and small system size.

The information system may itself serve as a reference coordinatesystem. However, the information system may solely represent anintermediate reference coordinate system in another reference coordinatesystem and that the relationship between the intermediate referencecoordinate system and the reference coordinate system is determined e.g.via the evaluation apparatus or another mechanism.

The signal input unit may include an optical signal unit that captureslight reflected back from the eye. Light is an excellent medium fortransmitting the signals reflected back from the eye since the presenceof light is a prerequisite for the ability to use the eye. However, theocular reflex signal information that results through the reflection onthe eye is superimposed with field-of-view signal informationtransmitted by the light from the field of view. These differing piecesof information, however, can be distinguished through use of knownsignal processing methods and can be sensibly used for determining theorientation of the eye. This is particularly true when the signaltransmission medium is from a signal source belonging to the informationsystem that applies a predetermined signal to the medium prior to itsreflection on the eye.

The signal input unit may also capture signals from other signaltransmission media. Components for generating and capturing sound waves,for example, are commercially available in various cost-efficient andcompact forms. Such components can also be implemented as integratedelements of an integrated circuit. Similar considerations apply to thenon-visible frequency range of electromagnetic waves.

The signal input unit may also capture signals from different media orspectral ranges thus providing improved system characteristics. This isbased on considerations that the evaluation unit takes over other systemtasks in the case of underload and that the signal processing carriedout by the evaluation unit depends strongly on the information contentof the signal to be processed. The information system can use signalcapturing that only demands a little performance from the evaluationunit, but itself might not supply the basis for sufficient accuracy andto complement and/or calibrate this low-processing signal capturing viathe results of an accurate and processing-intensive, yet onlyintermittently executed signal capturing such that the necessaryaccuracy is achieved at any time.

The capturing of the retinal reflex in which the retinal reflex ofnatural or artificial light is intermittently or partial captured as thesignal reflected back from the eye has turned out to be useful. A fullcapturing of the retinal reflex is both time consuming and demanding onperformance. On the other hand, a capturing of the retinal reflex isuseful inasmuch as it allows the relationship of the perceived field ofview to the retina to be directly determined. Indeed, as noted above, aprocessing of the captured retinal reflex allows both retinal featuressuch as e.g. the fovea centralis or the blind spot as well as the refleximage of the light falling onto the eye to be determined. The network ofblood vessels present in the choroid coat also becomes visible throughappropriate processing of the retinal reflex image, which yields a verygood basis for determining the orientation of the eyeball. If theretinal reflex is thus captured intermittently or partially, theprocessing complexity can be reduced without sacrificing an exactdetermination of the relationship of the perceived field of view to theretina. Naturally, the retinal features can be followed withoutcapturing the retinal reflex. For example, the blood vessels of thechoroid coat can be recognized via their radiation of heat that isvisible in the infrared range.

The above system may be used for the analysis of a patient's sight,wherein a predetermined pattern or a predetermined distribution ofpatterns is generated on the retina or on selected regions of the retinausing the projection unit. The system may use movement patterns and/orthe noise fields and/or the spatial vision of a patient's eye, whereinrandom dot patterns are generated on the retina using the projectionunit for test purposes. The system may be used for determining anomaliesin the motor response of the eyeball, wherein a unit for determining andmonitoring the position and/or orientation of the eyeball is integratedinto the system. The system may be used for detectingparasympathetic/sympathetic efferences, wherein the motor response ofthe pupil is monitored and evaluated by means of a detector device. Thesystem may also be used as one or more of the following: a synoptophoror synoptometer with no device convergence, a device for determiningcyclodeviation, a phase difference haploscope, a device for detectingphoria identical to the visual axis with different lines of sight, forchecking the function of the retina, making use of a sampleelectro-retinogram (ERG) and a correlation device, with which an imageplayed onto the retina can be brought into correlation with the ERGactually determined, for measuring the contrast sensitivity of apatient's sight, preferably as a function of the spatial frequency, forwhite-noise-field campimetry, for determining the extent and theposition of central field of vision defects (scotomae), as a VEP (visualenabling for precision surgery) device, or as an SLO (scanning laserophthalmoscope) device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows schematically an interactive data view and command systemin accordance with a first embodiment of the invention.

FIG. 2 is a detailed view of an eye in cross-section.

FIG. 3 shows schematically an embodiment of interactive spectaclesemployed in the invention in which an optical signal unit is provided inthe form of a scanning eye detection apparatus.

FIG. 4 shows schematically is an embodiment of interactive spectaclesemployed in the invention in which an output apparatus in the form of ascanning projection apparatus is provided.

FIG. 5A shows schematically interactive spectacles in accordance with afourth embodiment.

FIG. 5B is a detailed drawing of a combined signal capturing andprojection unit illustrated in FIG. 5.

FIG. 6A shows schematically interactive spectacles in accordance with afifth embodiment.

FIG. 6B is a detailed drawing of a combined signal capturing andprojection unit illustrated in FIG. 6A.

FIG. 7A shows schematically interactive spectacles in accordance with asixth embodiment.

FIG. 7B is a detailed drawing of a combined signal capturing andprojection unit illustrated in FIG. 7A.

FIG. 8 shows schematically interactive spectacles in accordance with aseventh embodiment.

FIG. 9 shows schematically interactive spectacles in accordance with aneighth embodiment.

FIG. 10A shows schematically interactive spectacles in accordance with aninth embodiment.

FIG. 10B is a front view of spectacles in accordance with a ninthembodiment.

FIG. 11A illustrates the naturally perceived field of view of a user ofan information system designed in accordance with a tenth embodiment.

FIG. 11B illustrates the naturally perceived field of view of a user ofan information system designed in accordance with a tenth embodiment.

FIG. 11C is a schematic representation of a scan pattern.

FIG. 11D is a schematic representation of a modified scan pattern.

FIG. 12A illustrates the naturally perceived field of view of a user ofan information system designed in accordance with an eleventhembodiment.

FIG. 12B illustrates the naturally perceived field of view of a user ofan information system designed in accordance with an eleventhembodiment.

FIG. 12C illustrates the naturally perceived field of view of a user ofan information system designed in accordance with an eleventhembodiment.

FIG. 12D illustrates the naturally perceived field of view of a user ofan information system designed in accordance with an eleventhembodiment.

FIG. 12E illustrates the naturally perceived field of view of a user ofan information system designed in accordance with an eleventhembodiment.

FIG. 13A is the naturally perceived field of view of a user of aninformation system designed in accordance with a twelfth embodiment.

FIG. 13B is the naturally perceived field of view of a user of aninformation system designed in accordance with a twelfth embodiment.

FIG. 14A shows schematically an information system in accordance withthe invention in accordance with a thirteenth embodiment.

FIG. 14B shows schematically an information system in accordance withthe invention in accordance with a thirteenth embodiment.

FIG. 15 is an information system in accordance with the invention inaccordance with a fourteenth embodiment.

FIG. 16 is a schematic representation of an information system inaccordance with the invention in accordance with a fifteenth embodiment.

FIG. 17 is an optical system in accordance with a sixteenth embodiment.

In the description of the figures, similar or identical objects aredesignated with similar or identically ending reference signs. Many ofthe illustrated objects comprise symmetrical or complementary componentsthat are distinguished via a supplementary letter, e.g. “L” for left and“R” for right, after the reference sign. If the statement applies toeach individual component of such a symmetrical or complementarygrouping, the supplementary letter is left out in some cases for thesake of clarity.

DETAILED DESCRIPTION

FIG. 1 illustrates schematically an interactive data view and commandsystem 100 in a broadest sense of the word, as an information system.The information system 100 is embodied in the form of an interactivespectacle system 120, i.e. interactive spectacles 120, which comprisetwo optical apparatuses 150. Preferably, the optical apparatuses 150 arerespectively located in an inner side of a left 121L or right 121Rtemple of the spectacles 120. Depending on the field of application,other arrangements of the optical apparatuses that do not disturb theview, e.g. in the region of a bridge 122 of the spectacles 120 thatcrosses the root of the nose of a user are also appropriate.

The optical apparatus is connected to a processor unit 140 viaconnection lines 101. If the optical apparatuses comprise photodetectorsand/or light sources, the connection lines serve the transmission ofelectrical detection and/or control signals. The photodetectors and/orlight sources can, however, be located in the processor unit 140 and beconnected to the optical apparatuses 150 of the spectacles 120 vialight-conducting connection lines 101. This contributes to reducing theweight of the spectacles 120.

Still referring to FIG. 1, a communication interface 196 (indicated bydash-dots) to which corresponding output signals from the signalprocessing unit 140 are transmitted in order to establish acommunicative connection with an external information unit 198 via apreferably mobile communication unit 197 that is connected to theinterface 196 via a signal line 196 a. In the most general case, theexternal information unit 198 is a database or data files that can beaccessed via suitable protocols and/or over the Internet.

A portable telephone, a laptop or a palmtop can, for example, serve as amobile communication unit 197, wherein the connection for the remotedata transmission can be carried out on the basis of all typicalprotocols such as GSM, UMTS, CDMA, TDMA or DECT. For example, a TDMAcommunication method and apparatus is described in U.S. Pat. No.6,160,800, which is incorporated by reference. A CDMA communicationmethod and apparatus is described in U.S. Pat. No. 5,878,036, which isincorporated by reference. Another mobile telecommunication system isdescribed in PCT Application PCT/SE99/02013 (published as WO 00/28769,which is incorporated by reference. A technique for providing a securelink in a mobile communication system is described in U.S. Pat. No.6,301,479, which is also incorporated by reference.

The signal line 196 a is bi-directional so that corresponding requestsignals can be returned via the signal line, as an interactive dataexchange between the communication unit 197 and the database 198, to theinterface 196 and, from there, to the signal processing unit 140. Acontrol apparatus, not shown in detail and preferably integrated intothe processor unit 140, ensures that the request signals are convertedinto the desired operation signals, with which either the optical unit150 is instigated to place further image information onto the retinaand/or at least a further information reproduction device, such as e.g.a headphone unit 150 a, is instigated to transmit additional informationto the operator. When a further information reproduction system 150 a isprovided, a further signal line 101 preferably exists that is lead tothe processor unit 140.

A microphone 150 a is indicated by a dash-dotted line that is acomponent of the data view and command system and via which speech inputcan be carried out. A corresponding signal line 101 b is lead from thespectacle system to the processor unit. Speech-controlled operation ofthe interactive data view and command system can be carried out via themicrophone. For example, control operations such as paging throughdisplay menus can be executed or particular system actions such as e.g.the triggering of select or deselect operations can be triggered.

The information system is capable of processing data from the opticalunit and the communication unit on the fly and simultaneously and, asthe case may be, as a function of additional control commands from theoperator that are either supplied to the processing unit via the line101 or the line 101 b, control the signal processing set-up 140 via thecommunication interface  and/or the further information reproductionunit 150 a such that the desired result of the interactive communicationis obtainable. It is thus, for instance, possible to convey acoustic orimage information to the operator that has been downloaded from theInternet via the communication unit upon gaze or verbal command. In thecase of optical information, said is superimposed, precisely positioned,onto image falling onto the eye.

A further possibility of controlling the data view and signal processingset-up—implemented via the components 140, 196, 197—consists e.g. ofplacing a control menu onto the operator's retina via the optical unit150. Since the image on the retina is captured cyclically, preferablywith a sampling frequency of 100 Hz, the optical unit 150, inconjunction with the processor unit 140, can determine, at any point intime, which image is situated at the center of the field of vision sincethis image is held in the fovea centralis. When the operator thusfocuses on a particular menu item or a blended-in program control symboland when, at this point in time, a particular trigger signal isgenerated, the corresponding operation is called. The trigger signal canbe generated by a suitable button provided on the system, via the speechinput system (microphone 150 b in connection with a speech decoder inthe processor unit) or else in an optical manner, e.g. by using a blinkof an eye consciously executed at that moment as a trigger.

Naturally, modifications of the system are conceivable without leavingthe spirit of the invention. Consequently, the mobile data transmissionset-up 197 can naturally be combined with the interface 196 and thesignal-processing set-up 140 to a unit, as indicated by the dash-dottedline Z.

The data transmission set-up can comprise a preferably portable computersuch as e.g. a laptop or palmtop that is equipped with a suitable remotedata transmission interface. An electromechanical control unit such ase.g. a control mouse can also be provided as a control unit foractivating particular operations of the system.

Since the optical apparatus constantly samples the operator's retina, itcan capture the retina's structure—when a suitable wavelength band, forinstance the infrared band, is chosen for the scan beam—so that uniqueassociable user/wearer ID data relating to the operator can beintermediately stored as a data record in the signal processing unit140. The data record can then be employed for wearer identification/forpersonalized set-up of the system to the respective operator.

The system is suited, in a particularly advantageous manner, for use inthe medical field, in particular in the field of ophthalmology, as atherapeutic or analytic device, as well as in military applicationsrelating to accessing large amounts of data in a mobile environment andto using corresponding data downloaded from the database for theoperator as contemporaneously as possible.

In the following, various embodiments of the optical apparatus aredescribed that can be used in an advantageous manner in combination withthe data view and command system in accordance with the invention.Furthermore, information systems are described that contain various,equally applicable, modifications of the optical unit and that aresensibly combinable with the data view and command system. For betterunderstanding of the manner of operation of the optical apparatus andthe system combined therewith, however, attention will first be given toFIG. 2 and, based on this representation, the structure of the human eyewill be elucidated.

To fully understand the information system 100, FIG. 2 provides adetailed view of an eye 280 in cross-section. The eye 280, which issituated in two eye sockets 20 (lat. orbita) formed of skull bone in thehead of a person and which is to be understood here in the sense of aneyeball 280, consists of a chamber surrounded by a translucent cornea283 and a visibly white sclera 28. The sclera 28 is covered on its sidefacing the inside of the eye 280 by a choroid coat 287 that supports,also on its inner side, a light-sensitive retina 281 and supplies samewith blood. Due to its pigmentation, the choroid coat 287 hinders ascattering of the light falling thereon that could disturb visualacuity.

The tissue of the retina 281 comprises two types of photoreceptor cells,i.e. rods and cones (not shown), which provide the person with theability to see. These photoreceptor cells absorb the light focuses by aneye lens 282 in a range of wavelengths from roughly 380-760 nm andconvert it, via a chain of chemical reactions, into electric nervesignals. The signals of the various nerve cells of the retina 281 arethen passed on to the brain via an optic nerve 25 and processed there toa perceptible image. The numerous, highly light-receptive rods, roughly120 million in number, are specialized for signal detection in twilight(so-called scotopic vision) and yield a grey-scale image. The roughly6.5 million, comparatively less light-receptive cones, in comparison,are responsible for color vision during daylight (so-called photopicvision). During light absorption, an oxidation of pigments in thephotoreceptor cells takes place. For the regeneration of the pigments,the cones require roughly six minutes and the rods require roughly 30minutes. An observation period of roughly 200 milliseconds is necessaryuntil the visual stimulus via the photoreceptors sets in and a receptionof information via the retina 281 takes place.

The retina 281 comprises a depression 286 that appears somewhat morestrongly pigmented due to its higher density of cones in comparison withthe rest of retina. This depression 286, which is typically called thefovea centralis, lies in a region of the retina known as the macula andrepresents the region of keenest vision. The fovea centralis 286 is onlyoccupied by cones, comprises a very high cone density and encompassessolely roughly 0.01% of the retina surface. The optic nerve 25 entersinto the inside of the eye via a sieve-like opening in the sclera 28 inan area vis-á-vis the lens 282 designated with the reference sign 288.This area 288 does not comprise any photoreceptor cells, whence it isnamed “blind spot.”

The chamber formed by the cornea 283 and the sclera 28 is partitioned bya deformable lens 282 and muscular ciliary processes 23 that supportsthe lens 282. The portion of the chamber lying the lens 282 and theretina 281, which makes up roughly two-thirds of the eyeball, forms aso-called vitreous humor 21, a gelatinous structure that consists, toover 98%, of water and that supports and protects the retina 281. Theportion of the chamber lying between the cornea 283 and the lens 282carries the name anterior chamber 22 and contains a fluid that nourishesthe cornea 283. In its neutral shape, the lens 282 typically refractsthe light falling on the eye such the far-away field of view is sharplyimaged onto the retina 281. Through contraction/relaxation of themuscles of the ciliary processes 23, the shape and thus also therefractive characteristics of the lens 282 can be varied over a widerange in order to allow e.g. a sharp imaging of close-lying objects inthe field of view onto the retina 281. In most cases, the personaffected is unaware of this process.

An aperture 285 of variable diameter and consisting of colored tissueregulates the light falling onto the light-sensitive portions of the eye280 and gives the eye 280 its characteristic coloring. Thus aperture 285is located in the anterior chamber 22 directly in front of the lens 282and is called the iris 285. Due to the low amount of light backscatteredby the lens 282, the vitreous humor 21 and the retina 281, the centralregion of the iris 285 appears black and called the pupil 284. Theregulation of the pupil size is also carried out by the personsubconsciously.

The eye 280 is connected to the skull via six muscles 24 that runpartially parallel and partially oblique to one another that allow apivoting of the eye 280 and subsequently a change of the gaze direction.The binocular field of view captured without moving the eyes 280encompasses roughly 170° horizontally and roughly 110° vertically. Ifthe eyes 280 are moved, a binocular field of vision of roughly 290°horizontally and roughly 190° vertically can be captured. The region ofkeenest vision captured by the fovea centralis 286 encompasses solelyroughly 1°. A fictitious axis through the center of this region iscalled the visual axis and corresponds to the direction of gaze. Arotation of the eye around the visual/optical axis is also enabled viathe muscles 24.

The six muscles 24 are responsible for all movements of the eye. Duringobservation of a fixed point, so-called micro-tremors of the eye 280take place, during which the eyes 280 tremble lightly in order to avoida temporary exhaustion of the ability of the affected photoreceptorcells to chemically react to a persistent stimulus. During a change ofthe direction of gaze or a movement of the head, so-called saccadicmovements take place, with whose aid the fovea centralis 286 is directedto its new target of fixation or held on its previous target offixation. During these highly complexly structured movements, the eye280 is involuntarily moved back and forth at a small amplitude of up toseveral tens of degrees and with an extremely fast angular velocity ofup to several hundred degrees per second. During the tracking of amoving object, the eye 280 achieves angular velocities of only one totwo hundred degrees per second.

To protect the eyeball 280, people have a movable fold of skin, i.e. anupper lid 27 a and a lower lid 27 b that allow a closing of the eyesocket 20 against external influences. The lids 27 a and 27 b close as areflex in the presence of foreign objects or strong light. Moreover, thelids 27 a and 27 b provide, via regular, typically involuntary blinking,for an evenly distributed film of tears on the cornea 283 that washesthe outer surface of the cornea 283 and protects it from drying out. Thelids 27 a and 27 b also comprise lashes 27 c that also protect the eye280 from dust. A conjunctiva 26 covers the space between the lids 27 a,27 b, the eye socket 20 and the eyeball 280. The conjunctiva 26 merges,on the one hand, with the inner side of the lid and, on the other hand,with the cornea 283 and represents a second wall of protection againstthe penetration of germs and foreign objects.

FIG. 3 shows schematically a first embodiment of the optical apparatusof the, as described above, interactive spectacles system/spectacles 320as a component of the interactive data view and command system. A signalcapturing apparatus in the form of a scanning eye scanning apparatus350D is provided. The left half of FIG. 3 represents a plan view ontothe head of a user 302 together with spectacles 320 having a righttemple 321R, whereas the right half of FIG. 3 reflects a cross-sectionof the spectacles 320 running through the left temple 321L. No furthercomponents of the information system 100 are shown in FIG. 3 other thanthe apparatuses belonging to the interactive spectacles 320.

In accordance with the illustrated embodiment, light beams 333 a and 333b falling onto the eye 380 that originate e.g. from the field of vieware sharply imaged onto the retina 381 by the lens 382 as a correlatedimage and are reflected back by the retina 381 as a retinal refleximage. A light beam 331 that has been reflected back in this mannerpasses again, in the opposite direction, through the lens 382, isfocused via two concave mirrors 322 and 323 that belong to the mirrorsystem of the spectacles 320 and is directed, as shown, onto a scanningeye scanning apparatus 350D. The eye scanning apparatus 350D comprises asignal capturing apparatus 351 in the form of a photodetector 351 thatcaptures the light beam 331 reflected back from the retina 381 as wellas two movable flat mirrors 352H and 353V that effect ahorizontal/vertical deflection of the light beam 331 onto thephotodetector 351. In accordance with the embodiment of FIG. 3, thespectacles 320 additionally comprises a light trap 324 that prohibits anincidence of light from undesired directions of incidence. To simplifythe mirror system of the spectacles 320, this mirror 323 can beimplemented via a mirrored inner surface of the spectacle lens. However,the surface must have a particular shape in order to facilitate acapturing of the entire retinal reflex image even when the eye 380 ispossibly in a skewed position. This, on the other hand, limits thedesign freedoms of the spectacles 320.

A serial, point-by-point scanning of the retinal reflex image as a pixelsequence is carried out via the combination of a point-shaped detector351 and corresponding control of the flat mirrors 352H and 352V.Preferably, the retina 381 is scanned with a circular, spiral orelliptical scan pattern as described in the PCT ApplicationPCT/EP97/04188 (published as WO98/05992) and U.S. application Ser. No.09/462,440. This has the advantage that the flat mirrors 352 can bedriven without jerky movements and that a higher pixel density (numberof pixels per unit of area of the retina) can be captured in the regionof the fovea centralis 286.

Preferably, a suitable synchronization operation for determining thecurrent optical axis is carried out prior to the capturing operation—tothe respect that it has not already been carried out in a previousprojection operation—so that the scan operation can be carried outcentered to the eye.

FIG. 4 shows schematically an embodiment of the interactive spectacles420 in which an output apparatus is in the form of a scanning projectionapparatus 450P. The left half of FIG. 4 represents a plan view onto thehead of a user 402 together with spectacles 420 having a right temple421R, whereas the right half of FIG. 4 reflects a cross-section of thespectacles 420 running through the left temple 421L. No furthercomponents of the information system 100 are shown in FIG. 4 other thanthe apparatuses belonging to the interactive spectacles 420.

In accordance with the illustrated embodiment, a scanning projectionapparatus 450P comprises a light source 453, e.g. a laser diode or anLED focused via a lens system that emits a projection light beam 432 aswell as two movable flat mirrors 454H and 454V. The projection lightbeam 432 is directed via the movable flat mirrors 454H and 454V onto amirror system of the spectacles 420 that comprises two concave mirrors422 and 423 that projects the projection light beam 432 onto the lens482 of an eye 480 and, in the end, onto the retina 481. To simplify themirror system of the spectacles 420, this mirror 423 can be implementedvia a mirrored inner surface of the spectacle lens. However, the surfacemust have a particular shape in order to facilitate a capturing of theentire retinal reflex image even when the eye 480 is possibly in askewed position. This, on the other hand, limits the design freedoms ofthe spectacles 420. To avoid the incidence of light that would bedisturbing, the spectacles 420 can be equipped with a light trap 424that hinders the incidence of from undesired direction of incidence.

A serial, point-for-point projection of an image is carried out via thecombination of a point-shaped light source 553 with correspondingcontrol of the flat mirrors 452H and 452V that respectively effect ahorizontal/vertical deflection of the projection light beam 432. Theprojection is preferably carried out, as described in U.S. Pat. No.6,227,667 and U.S. application Ser. No. 09/462,440, in a circular,spiral or elliptical scan pattern. This has the advantage that the flatmirrors 452 can be driven without jerky movements and that a higherpixel density can be projected onto the retina 481 in the region of thefovea centralis 286.

The degree of perception of an image projected into the eye 480 can becontrolled in relation to the naturally perceived image via thebrightness of the projected pixels. However, retinal perception is ahighly complex process in which psychological effects also play a strongrole. In this respect, reference is made to the relevant literature ofthe field.

In simplified form, however, one can say that the retina 481 adapts tothe brightness of the total light falling thereon. It is known, forexample, that the slight glow of the clock of a radio alarm that cannoteven be perceived in daylight can appear to illuminate an entire room ina dark night. On the other hand, the strong light of headlights ofapproaching vehicles is barely perceptible in daylight. The brightnessof a single pixel is thus perceived in relation to the pixels otherwiseperceived. The retina 481 functions similarly when observed locally. Ifthe brightness of a pixel projected onto a region of the retina 481exceeds the brightness of the light otherwise falling onto this regionby roughly 10%, then solely the projected pixel is effectively perceivedby this region of the retina 481 in lieu of the other light. Due topsychological effects, the exact value can also lie between 5%-10%,10%-15% or even 15%-20% instead of at 10%.

Preferably, a suitable synchronization operation for determining thecurrent optical axis is carried out prior to the projection operation—tothe respect that it has not already been carried out in a previousscanning operation—so that the projection operation can be carried outcentered to the eye.

FIG. 5A shows schematically interactive spectacles 520 in accordancewith a fourth preferred embodiment in which a combined signal captureand projection apparatus 550 is attached to the spectacles 520 in theregion of the bridge 522. Referring to FIG. 5B, the combined signalcapture and projection apparatus 550 comprises both a projectionapparatus 553 as well as a signal capturing apparatus that are housedtogether in a protective housing 558. Light beams 530 make their wayinto the inside of the housing 558 and vice-versa via a translucentwindow 559 in an outer wall of the housing 558. The sealing of thehousing 558 via the window 559, however, keeps dust, sweat and otherforeign materials from disturbing operation of the combined signalcapture and projection apparatus 550.

Light beams 530, 530 a, 530 b are captured/projected analogously to thedescribed systems in accordance with FIGS. 3 and 4. The interactivespectacles 520 can be simplified, however, in their construction byreplacing the mirrors 352/452, which are separate in the prior art, forvertical/horizontal deflection of the respective light beams 331/432with a swiveling mirror 552/554. For the purpose of achieving a compactdesign, a partially transmissive mirror 556 can serve to allow separatebeam paths within the housing 558 for the light 530 falling or projectedthrough the window 559. Preferably, the inner side of the spectacle lensis provided with a surface 523 that strongly reflects beams incidentfrom this direction that is used as a mirror for the beam path betweenthe eye 580 and the combined signal capturing and projection apparatus550. This contributes to a reduction of the components required andresults, in the illustrated embodiment, in a simplifiedtransmission-efficient beam path 530 in which the light beam 530 betweenthe eye 580 and the projection/signal capturing apparatus 553/551 isonly reflected three times. As described above, however, this results ina limitation of the design freedoms of the spectacles 520.

The freedom of movement necessary for a swiveling motion of the mirror552, 554 can be achieved, for example, via a Cardan joint or springsuspension of the mirror 552, 554. Possible embodiments of such aswiveling mirror are known to the person skilled in the art, e.g. fromthe field of microtechnology. Further solutions to the presentdeflection problem in which the respective light beam 530 is deflectedon the basis of electrochrome, holographic, electro-holographic or otherlight refraction or light reflection mechanisms are easily conceivableand equally applicable.

Although the interactive spectacles 520 are shown in a minimalistembodiment in which a combined signal capturing and projection apparatus550 is solely provided for the left eye 580, it is self-evident that asecond combined signal capturing and projection apparatus 550 having amirror image design can be provided for the non-illustrated right eye,if necessary, in the region of the right half of the bridge 522.

FIG. 6A shows schematically a modification of the spectacles 520illustrated in FIGS. 5A and 5B. Interactive spectacles 620 in accordancewith a fifth preferred embodiment utilize the left combined signalcapturing and projection apparatus 550L situated in the region lyingbetween the left spectacle lens 624L and the left temple 621L and theright combined signal capturing and projection apparatus 650R situatedin the region lying between the right spectacle lens 624R and the lefttemple 621R.

Such an arrangement of the combined signal capturing and projectionapparatuses 650L, 650R vis-á-vis the respective spectacle lenses 624L,624R and the respective eyes 680 is typically associated with thenecessity of either providing several mirrors along the beam path 630(cf. mirror 322 and 323 in FIG. 3) or bestowing the respective spectaclelens 624L, 624R with a particular form in order to guarantee a captureof all regions of the retina 681. This, however, significantly limitsthe design freedoms of the spectacles 620. In order to circumvent thisproblem, the interactive spectacles 620 in accordance with FIG. 6propose spectacle lens 624L, 624R whose inner sides are provided with arespective holographic coating 623L, 623R. Such a holographic coating623 is capable of emulating an arbitrary reflection topology. Forexample, a holographically coated, flat surface can act like aspherically curved surface. Similarly, a holographically coated,spherically curved surface can act like a flat surface. The change ofthe effective reflection topology depends solely on the holographiccontent of the coating. In accordance with the Figure, the holographiccoating 623L and 623R are designed and situated as mirror images to oneanother.

FIG. 6B shows schematically the combined signal capturing and projectionapparatus 650L. Analogously to the combined signal capturing andprojection apparatus 550, illustrated in FIG. 5B, apparatus 650Lcomprises a housing 658, a projection apparatus 653 and a signalcapturing apparatus 651, respective swiveling mirrors 652 and 654, apartially transmissive mirror 656 and a housing window 659.

FIG. 7A shows schematically a modification of the spectacles 520 shownin FIGS. 5A and 5B. Interactive spectacles 720 in accordance with asixth, preferred embodiment utilize the left combined signal capturingand projection apparatus 750L situated in the region lying between theleft spectacle lens 724L and the left temple 721L and the right combinedsignal capturing and projection apparatus 750R situated in the regionlying between the right spectacle lens 724R and the left temple 721R.

FIG. 7B shows schematically the combined signal capturing and projectionapparatus 750L. Analogously to the combined signal capturing andprojection apparatus 550 illustrated in FIG. 5B, it comprises a housing758, a projection apparatus 753 and a signal capturing apparatus 751,respective swiveling mirrors 752 and 754, a partially transmissivemirror 756 and a housing window 759.

The problem of the beam path 730 touched upon above is solved in thisembodiment in a space-saving manner via a special design of pads 725Land 725R. Typically, spectacles 720 are supported on the root of thenose either through the bridge 722 or through so-called pads 725. Intheir typically commercial design, pads are relatively flat, slightlycurved and oval. Moreover, they are either hingably or swivably mountedon a projection extending from the bridge 722 in order to ensurecomfortable contact of the pads to the side surfaces of the root of thenose. In the illustrated embodiment, the pads 725 are formed asfixed-shaped, elongated units that project from the spectacles 720 inthe direction of eye 780 in the region of the bridge 722. On theirrespective longitudinal sides facing the nose, the pads 725 form thecontact surfaces that support themselves on the root of the nose. Intheir end region lying across from the spectacles 720, the pads 725comprise a support surface on the respective side facing the eye that isprovided with a mirror or a mirroring coating, e.g. a metallic coatingor a holographic coating.

Although the frame of the spectacles 720, including the pads 725, has aprincipally fixed form, both quasi-static, e.g. due to material fatigueand/or temperature changes, as well as dynamic deformations of theframe. Particularly when the spectacles 720 are put on and duringactivities in which vibrations are commonplaces, changes to the relativearrangement of the respective spectacle components to one anotherresult. The relative location of the spectacles 720 vis-á-vis the eye780 is also not constant. Accordingly, both the optical system of thespectacles 720, i.e. those system components that contribute to theoptical signal capturing and/or the optical projection as well as anyprocessing system connected to thereto, must be conceived and designedsuch that such changes in the arrangement can be taken intoconsideration and/or compensated and/or do not cause any extraordinaryoperational disturbances. This also holds for all types of interactivespectacle systems.

The problem addressed above can be overcome in particular through asuitable signal processing of the captured signals and the signals to begenerated. Furthermore, an optical marker fixed disposed on thespectacle frame in the vicinity of the typical beam path 730 can beadditionally detected on a regular basis or on demand via the signalcapturing apparatus 751 for the purpose of calibrating its opticalsystem.

FIG. 8 shows schematically a modification of the spectacles 520illustrated in FIGS. 5A and 5B, interactive spectacles in accordancewith a seventh preferred embodiment in which the signal capturingapparatus 851 of the combined signal capturing and projection apparatus850 is capable of at least partially capturing the corneal reflex image.

The cornea is typically formed rotationally symmetric to the opticalaxis. Beams that fall perpendicularly onto a central region of thecornea are thus confocal to the optical system of the eye 880 and formthe basis of the image truly perceived on the retina 881. Moreover, thecornea 883 consists, to a large degree, of water and exhibits, for thisreason, a very high degree of reflectivity at a wavelength of roughly1.1 μm. Since this wavelength lies in the infrared spectral region, acapturing of the corneal reflex image is primarily suitable for infraredapplications, e.g. for night vision devices. Reflections occur not onlyon the outer, concave corneal surface, however, but also on the insideof the cornea. Moreover, due to its structure, the cornea 883 does noteffect mirror-like reflections but instead effects a diffuse reflectionthat becomes more diffuse with increasing depth of the act of reflectionwithin the cornea.

In order to obtain a meaningful corneal reflex image, effectively onlythose beams that fall perpendicularly onto a central region of thecornea are captured in the illustrated embodiment. This is achievedthrough several measures. Firstly, the spectacle lens 824 situated infront of the eye whose side facing the eye 880 is provided with asurface 823 that is high reflective for beams incident from thisdirection comprises a specially designed shape that focuses the lightperpendicularly reflected from the cornea such that it falls onto thesignal capturing apparatus 851 as light beams 834 that run nearly inparallel whereas light that is reflected non-perpendicularly from thecornea is deflected in another direction. Alternatively, the spectaclelens 824 can be designed in another fashion, yet comprise a partiallytransmissive, holographically reflecting layer 823 that likewise effectssuch a focusing of the light reflected perpendicularly from the corneasuch that it falls onto the signal capturing apparatus 851 as lightbeams 834 that run nearly in parallel, whereas light non-perpendicularlyreflected by the cornea is deflected in another direction. Secondly, anaperture 857 is provided shortly in front of the signal capturingapparatus 851 that prohibits a capturing of those light beams whoseincident angle lies outside a narrow range of incident angles of thelight beams 834 that run nearly in parallel as described above.

FIG. 8 shows schematically a modification of the spectacles 520illustrated in FIGS. 5A and 5B. The interactive spectacles in accordancewith another preferred embodiment utilize the signal capturing apparatus851 of the combined signal capturing and projection apparatus 850capable of at least partially capturing the corneal reflex image.

The cornea is typically formed rotationally symmetric to the opticalaxis. Beams that fall perpendicularly onto a central region of thecornea are thus confocal to the optical system of the eye 880 and formthe basis of the image truly perceived on the retina 881. Moreover, thecornea 883 consists, to a large degree, of water and exhibits, for thisreason, a very high degree of reflectivity at a wavelength of roughly1.1 μm. Since this wavelength lies in the infrared spectral region, acapturing of the corneal reflex image is primarily suitable for infraredapplications, e.g. for night vision devices. Reflections occur not onlyon the outer, concave corneal surface, however, but also on the insideof the cornea. Moreover, due to its structure, the cornea 883 does noteffect mirror-like reflections but instead effects a diffuse reflectionthat becomes more diffuse with increasing depth of the act of reflectionwithin the cornea.

In order to obtain a meaningful corneal reflex image, effectively onlythose beams that fall perpendicularly onto a central region of thecornea are captured in the illustrated embodiment. This is achievedthrough several measures. Firstly, the spectacle lens 824 situated infront of the eye whose side facing the eye 880 is provided with asurface 823 that is high reflective for beams incident from thisdirection comprises a specially designed shape that focuses the lightperpendicularly reflected from the cornea such that it falls onto thesignal capturing apparatus 851 as light beams 834 that run nearly inparallel whereas light that is reflected non-perpendicularly from thecornea is deflected in another direction. Alternatively, the spectaclelens 824 can be designed in another fashion, yet comprise a partiallytransmissive, holographically reflecting layer 823 that likewise effectssuch a focusing of the light reflected perpendicularly from the corneasuch that it falls onto the signal capturing apparatus 851 as lightbeams 834 that run nearly in parallel, whereas light non-perpendicularlyreflected by the cornea is deflected in another direction. Secondly, anaperture 857 is provided shortly in front of the signal capturingapparatus 851 that prohibits a capturing of those light beams whoseincident angle lies outside a narrow range of incident angles of thelight beams 834 that run nearly in parallel as described above.

FIG. 9 shows schematically a modification of the spectacles 520illustrated in FIGS. 5A and 5B. The interactive spectacles in accordancewith another preferred embodiment utilize a spherical orspherical-acting, partially transmissive, mirroring supplementaryelement 929 arranged between the spectacle lens 924 and the eye 980.Preferably, the supplementary element 929 is arranged confocal to theoptical system of the eye 980.

The degree of reflectivity of the supplementary element 929 can beadapted to the requirements of the information system. One can choosebetween a high degree of reflectivity, which allows very good capturingof light beams 933 a-933 c directed onto the eye 980, and a low degreeof reflectivity, which avoids impairment of the perception carried outby the eye 980. Preferably, the supplementary element 929 exhibits a low(e.g. less than 10%), homogenous degree of reflectivity over its entirereflective surface. On the other hand, reflecting organs of the eye 980,for instance the cornea 983 or the retina 981, exhibit, in part, verystrong local reflective dependencies. Similar statements hold for thespectral reflective dependencies of the supplementary element and/or thereflecting organs of the eye 980. Whereas the supplementary element 929can be preferably designed such that it exhibits a homogeneous degree ofreflectivity over all relevant spectral ranges, the various organs ofthe eye 980 exhibit highly differing degrees of absorption that, in manycases, are also subject to strong local variations.

Excepting partial reflection, the supplementary element 929 should haveas little effect as possible on the light falling thereon. For thisreason, the supplementary element 929 is preferably manufactured of ahomogenous, translucent and uncolored material and is manufactured tohave a constant thickness in the direction of the light beams directedtoward the center of the eye. By applying an anti-reflective coating tothe side of the supplementary element 929 facing the eye 980, improvedtranslucency can be achieved.

The reflecting contour of such a supplementary element 929 is welldefined and can thus be supplied to the information system as knowninformation, whereas the contour of the relevant reflecting organs ofthe eye 980 must first be determined. In some respects, latterencompasses significant difficulties. The capturing of the light beams933 a-933 c directed onto the eye 980 can thus yield valuable images ofthe field of vision.

In the illustrated embodiment, effectively only those beams that fallperpendicularly onto the supplementary element 929 are captured. This isachieved through the following measures:

Due to the partially reflective surface of the supplementary element929, a corresponding portion of those beams 933 a-933 c that fallperpendicularly onto the surface of the supplementary element 929 arereflected back perpendicularly, whereas other beams are reflected backfrom the surface of the supplementary element 929 correspondingly skewedin accordance with the law of reflection “The angle of incidence equalsthe angle of reflection.” The light beams reflected back perpendicularto the surface of the supplementary element 929 travel back the same waythey came and are thus incident upon the spectacle lens 924 situated infront of the eye. The side of the spectacle lens 924 facing the eye 980is provided with a surface 923 that is highly reflective for beamsincident from this direction and comprises a specially designed shape ora specially formed coating that focuses the light beams reflectedperpendicularly by the supplementary element such that they fall ontothe signal capturing apparatus 951 as light beams 934 that run nearly inparallel whereas light beams reflected non-perpendicularly by thesupplementary element are deflected in another direction. In addition,an aperture 957 is provided shortly in front of the signal capturingapparatus 951 that prohibits a capturing of those light beams whoseincident angle lies outside a narrow range of incident angles of thelight beams 934 that run nearly in parallel as described above.

If the image of the field of view captured via the supplementary element929 is to be the basis for a projection correlated with the actuallyperceived field of view, then the correlation between the captured lightand the perceived field of view must be determined. In accordance withthe embodiment of FIG. 9, this correlation is achieved through apreferably confocal arrangement of the supplementary element 929 to theoptical system of the eye 980. Thus, preferably, the supplementaryelement 929 is fastened to the spectacles via an adjustable suspensionsuch that the position of the supplementary element 929 can be adjustedin both vertical as well as in two horizontal directions.

To obtain confocal arrangement, the supplementary element 929 issituated rotationally symmetric to the optical axis and is spaced fromthe lens 982 such that the optical mid-point of the optical system ofthe eye agrees with the mid-point of the sphere defined by the sphericalor spherical-acting supplementary element. The optical axis can besufficiently determined for this purpose via the orientation of thepupil 984 that is easily recognizable via its sharp contours and whoseorientation is easily determinable due to its round shape. In addition,due to the spherical or spherical-acting shape of the supplementaryelement 929, no pivoting of the supplementary element 929 around thepossible pivotal axes of the eye 980 is necessary to ensure confocalitysince, even the case of a skewing of the eye, at least a substantialportion of the supplementary element 929 remains, in terms of optics,rotationally symmetric to the optical axis through a correspondingvertical and/or horizontal shift of the supplementary element 929. Asregards the distance to the lens 982, there are various possibilitiesfor determining the necessary distance. For example, an optical oracoustic measurement of the cornea 983 can be carried out whosecurvature yields a very good estimate of the correct location of thesupplementary element 929. Retinal or cornea reflex images can also beat least partially captured, and the correct distance can be determinedon the basis of a comparison of the reflex images with the lightcaptured via the supplementary element 929.

Due to the spherical or spherical-acting implementation (e.g. through aholographic coating) of the partially reflecting surface of thesupplementary element 929 as well as through the confocal arrangement ofthe supplementary element to the eye 980, solely those beams 933 a-933 cthat fall perpendicularly onto the surface of the supplementary element929 are confocal to the optical system of the eye 980 and thuscorrespond to the beams falling onto the retina.

FIG. 10A is a plan view and FIG. 10B is a front view of spectacles 1020in accordance with another embodiment utilizing two sensor devices 1061Rand 1061L. For example, two solid-state cameras (e.g., CCD or TTLcameras) are provided to capture additional signal, in particular in thevisible field of vision. FIG. 10B also shows the left eye 1080L and theright eye of a wearer 1002 wearing the spectacles 1020. For the sake ofclarity, however, no other features of the user 1002 are represented inFIG. 10B.

To avoid the occurrence of parallax between the images captured by therespective cameras 1061R, 1061L and the images received by the eyeassociated therewith, the cameras 1061 should be arranged as coaxiallyas possible to the eyes with regard to their “optical axes.” In view ofthe system size of such solid-state cameras 1061 and the current stateof the art, it has turned out to be meaningful to located the cameras1061 in the front region of the respective temples 1021L, 1021R asshown. A mounting in the region of the bridge 1022, e.g. in the pads1025, is also meaningful. After a further miniaturization, thesolid-state cameras 1061 will be able to be located in the spectacleframe over the respective spectacle lenses 1024L, 1024R in order toachieve further axial identity. It is foreseeable that solid-state andother types of light capturing systems will, in the future, be able tobe built into the spectacle lens 1024, which can naturally be glass,plastic or other translucent material. Such an arrangement of thecameras 1061 would allow a signal capturing that is coaxial and nearlyconfocal with the eye 1080L, 1080R.

In a non-coaxial arrangement of the sensor apparatus 1061 to therespective eyes 1080L, 1080R, the information obtained from the sensorapparatuses 1061 should be brought into correlation with the eyes 1080,as need be. Such correlation is particularly important when the sensorapparatuses 1061 are implemented by cameras 1061 and a superimposedimage based on image information obtained from the cameras 1061 is to beprojected into the respective eye 1080L, 1080R.

If the image captured by the cameras 1061 is simply projected on therespective eye 1080L, 1080R, so-called parallax occurs, in which the“field of view” of the respective camera 1061L, 1061R does not agreewith the naturally perceived field of view. In particular during askewing of the eye 1080 deviating from the neutral position, or in thecase of objects lying closer in the field of view, parallax would leadto abnormal perception in the case of superimposition since, in suchcases, the optical axis of the eye 1080 would lie skewed to the “opticalaxis” of the respective camera 1061L, 1061R.

During correlation in this sense, only the portion of the image capturedby the cameras 1061 is projected into the respective eye 1080L, 1080Rthat lies in corresponding “correlation” to the optical axis of therespective eye 1080L, 1080R. In the simplest case, an at least partialreflex image of the field of view is captured from the respective 1080L,1080R via the signal capturing apparatus 1051. Characteristic pixelsthat can be found in both the captured reflex image as well as in theimages captured by the cameras 1061 then serve as reference points for aperspectively correct projection of the image information captured bythe cameras 1061 onto the eye 1080. Similarly, the signals captured fromthe eye 1080 can serve to determine the gaze direction of the respectiveeye 1080L, 1080R with respect to the coordinate system of the spectacles1020 in order to carry out a mathematically based correlation from thisangular information.

The correlation is also meaningful in the context of system applicationsin which the eyes 1080 are hindered from perceiving the field of view.This is the case, for example, during use of occluded, so-called“virtual reality” glasses 1020 (as shown, yet with non-translucentlenses 1024) wherein solely a synthetically generated image is presentedto the eyes 1080. In such a case, the aforementioned correlation canconsist, for example, of capturing the gaze direction of the eye 1080 asdescribed above and projecting a virtually generated image thatcorresponds to the orientation of the respective eye 1080L, 1080R. Inthis case, however, the spectacles 1020 serve as a coordinate system.If, however, the position and orientation of the spectacles 1020 is alsodetermined, e.g. on the basis of the images captured by the cameras1061, then a correlation between the respective eye 1080L, 1080R and thesurroundings can be created. Such a system could be used, for example,in a virtual amusement house, similar to a house of horrors. Someonewho's currently standing on a conveyor belt could have, for example, avirtual image projected into the eyes that gives him the feeling he isrunning on floating tree trunks in the middle of a wild river.

We emphasize that the information system described above in connectionwith FIGS. 5 to 10 must not necessarily operate with a combined signalcapturing and projection apparatus. According to another preferredembodiment, the information system may include separate signal capturingand projection apparatuses. According to another preferred embodiment,the information system may include only a signal capturing apparatus,and according to yet another preferred embodiment, the informationsystem may include only a signal projection apparatus, or in anotherembodiment none of the two apparatuses. Any of the signal capturing orprojection apparatuses may execute only partial or limited capture orprojection.

FIGS. 11A and 11B illustrate schematically the use of information system100 a telescope for a user. FIG. 11A shows the naturally perceived fieldof view 1190 of a user. Although the field of view 1190 encompassesroughly 170° of the surroundings horizontally and roughly 110° of thesurroundings vertically, solely a small region 1191 of several degreesaround the visual axis forms the region of keenest sight 1191.

Via its capturing of light from the field of view and the aforementionedpossibility of a projection of image information into the eye, theinformation system can be designed such that this region 1191 isprojected, optically enlarged, onto the region of keenest sight 1191after corresponding processing of the captured pixels via an evaluationapparatus comprised by the information unit, e.g. upon pressing of abutton. As described above, the degree of perception of an imageprojected in this manner in relation to the naturally perceived imagecan be controlled via the brightness of the projected pixels. If thefield-of-view light is captured, for instance, as a reflex image fromthe eye, a spatial or temporal separation of the capturing and theprojection will ensure that the projection does not influence thecapturing.

In the case of a conventional telescope, the spatial relationship to thesurroundings is lost on account of the fact that the entire field ofview is shown in enlargement. As a consequence, a person looking througha telescope cannot walk or drive at the same time. This phenomena iswell known.

Since the information system 100 can determine, by capturing signalsfrom the eye, the visual axis/the position of the fovea centralisrelative to the optical system of the spectacles, the information systemis capable of avoiding this disadvantage of a conventional telescope.For example, the projection can be carried out in a manner shown in FIG.11B. The system projects a small region 1191 lying directly around thevisual axis in the natural field of view onto the fovea centralis inenlargement, whereas no projected image information is superimposed onthe remainder of the field of view. The scene peripherally perceived bythe user thus stays the same in spite of telescopic presentation of themost relevant region of the field of view. In order to achieve thiseffect, the brightness of the image information projected into the eyemust naturally be chosen such that the desired relationship ofperception between the natural and the projected image results. Thissystem also has the advantage that the amount of image processingnecessary for the enlargement is held within limits since only aselected image region 1191 of the field of view 1190 is processed.

In accordance with another embodiment, an enlarged image is projectedinto the eye such that the projected image in an annular border regionbetween the region of keenest sight 1191 and the remaining region of theretina is enlarged more strongly as it gets closer to the visual axis.In this case, no enlargement takes place along the outer edge and alongthe inner edge an enlargement takes place with the same “zoom factor” asthe enlarged image projected into the inside of the ring, i.e. onto thefovea centralis. Thus, when the brightness of the projected imageinformation is correspondingly chosen, a soft transition between theperipheral scene and that which is telescopically seen results.

FIGS. 11C and 11D schematically illustrate the enlargement of the imagenaturally falling onto the fovea centralis by modifying a scan pattern1138, 1139 during the scanning of a reflex image. In FIGS. 11C and 11D,the projection pattern 1137 and scan patterns 1138, 1139 are illustratedin the same plane, for the sake of explanation. In general, in theinformation system 100, the projection can take place onto the retina,whereas the scanning can takes place, for example, from the cornea, andthis may be done by different units (or even on different eyes of thewearer).

FIG. 11C schematically illustrates a typical scan pattern 1138 thatscans the region 1189 of the cornea or retina reflecting the field ofview. In this vastly simplified example, it is assumed, for the sake ofcomprehensibility, that the respective pixels of the sequentiallyscanned image are projected back, after image-processing preparation ifneed be, in their proper sequence as corresponding images of thesequential image projected into the eye. In the illustrated example, thescan pattern 1138 thus corresponds to the projection pattern 1137 inspite of possible spatial or temporal separation of the scan beam andthe projection beam. If an enlargement of a central region of the fieldof view is desired, then the scanning can be effected in accordance witha modified scan pattern 1139 that effects an increase in the density ofthe captured pixels in that central region. If these pixels captured athigher density are projected back correspondingly, yet at lower densityduring the projection, then an enlarged image is the result.

In accordance with another embodiment, information system 100 isconstructed and arranged as a guidance system. For this purpose, theinformation unit of information system 100 comprises position sensore.g. acceleration measurement apparatuses or GPS receivers as well as adata bank or data bank connection that supplies orientation data. Such adata bank can be implemented e.g. via a CD-ROM carrying the data, a DVDor another exchangeable storage medium in connection with acorresponding reading device. Methods and apparatuses for obtainingposition fixing information that, for example, determine the currentlocation or allow their determination via a combination of suchorientation data with data obtained from the position sensors are known.In a typical apparatus, the orientation data comprise map informationthat are used for position determination in conjunction with signalssupplied by the position sensors. The establishment of a correlation ora dependency, e.g. when such position information is obtained orpresented, between the signals captured from the eye or light capturedfrom the field of view and the provision of information, however, farexceeds that known to the art.

FIGS. 12A to 12E show the field of view 1290 perceived by a user of aninformation system designed in form of a guidance system, according toanother embodiment. The system evaluates the captured field-of-viewlight with respect to the positioning information obtained via a patternrecognition that takes the data available for the determined whereaboutsinto consideration. Orientation hints such as characteristic buildings,side streets, or the like that are to be expected for the determinedwhereabouts are thereby recognized such that e.g. visual or acousticguidance or identification can be carried out, if necessary. Theseoperations may be performed by the information unit.

In the illustrated example in accordance with FIG. 12A, the guidancesystem serves for navigation. In this case, it is determined, e.g. basedon a computed or predetermined route, available map information and thecurrent whereabouts, that one should turn into the second street on theright side. This street is recognized on the basis of the capturedfield-of-view light via pattern recognition, in response to which ahinting arrow pointing at the street is locationally correctly blendedinto the field of view via projection taking into consideration the gazedirection determined by the system. Similarly, the guidance system couldprovide the driver with an acoustic message, e.g. “turn right after 50meters” or “now turn right.”

In the example illustrated in FIGS. 12B and 12C, the guidance systemserves to provide information. For example, a user can selectively beprovided information about his direct surroundings. Referring to FIG.12B, a tourist using the information system looks at a characteristicbuilding and actuates an activation button that is physically present orvirtually blended into the field of view. The building is subsequentlyidentified on the basis of the determined whereabouts and a patternrecognition based on the captured field-of-view light or an electroniccompass that determines the direction of the head. In response,information about the building is provided. This information canoriginate from a data bank or other information source and can beselected, e.g. interactively, via context-dependent menu that visuallyor acoustically lists the information available for selection for thatparticular building. The selection can be carried out via voice controlor via fixation with the eyes. Further information re eye-controlledmenu selection will be described in a later section of this description.

Referring to FIG. 12B, historic data are blended into the field of viewvia projection. In doing so, the system determines, from the capturedfield-of-view light, a suitable blend-in position, e.g. in front of amonotonous roof or in front of the sky. The data are blended in inaccordance with the blend-in position. Typically, the fovea centralis isnot directed at the blend-in position at first, whence the blended-indata is first perceived as an unfocussed, peripheral appearance. Thelocationally fixed, blended-in data are not imaged upon the foveacentralis until a corresponding pivoting of the gaze direction inaccordance with FIG. 12C. If the gaze is directed at another buildingrecognized by the system, then the blend-in information can change inaccordance with FIGS. 12D and 12E. In the Figures, the circle 1290represents the perceived field of view, whereas the circle 1291designates the region of the field of view captured by the foveacentralis.

By using a compact and portable design shown in FIG. 1, the informationsystem 100 can be used as an orientation system worn by a pedestrian, abicyclist, a motorcyclist or other vehicle driver.

FIGS. 13A and 13B illustrate the use of the information system 100 as adriving aid, in accordance with another embodiment. Preferably, theinformation system 100 includes an information unit comprising adistance sensor (e.g. an optical or acoustic distance measuring device)or a radar apparatus. Alternatively, the information unit is connectedto an external distance measuring system. The distance sensor determinesthe distance between a vehicle and objects located in front of thevehicle in the direction of motion. In the case of a stereoscopiccapturing of light from the field of view, the distance could bedetermined via a computation of parallax in which change of position ofthe object in a respectively captured left and right image conveysinformation re the distance.

The information unit may also include an evaluation apparatus that candetermine a (or calculate a probability), that the vehicle is on acollision course with an object within the field of view. A user of thesystem has a perceived field of view 1390 (shown in FIGS. 13A and 13B).Upon determining a potential collision course, the driving aid displays,e.g., a warning symbol 1395, which can be blended in the region ofkeenest sight 1391 and a warning circle 1394 can be blended in aroundthe dangerous object by projection, as described above. If the object islocated outside or on the edge of the region of peripheral vision, thena further warning symbol 1395 a can attention to where the danger lurks,as shown in FIG. 13A.

Other information relevant to driving safety can also be determined viasensors or the captured field-of-view light. For example, an evaluationapparatus could recognize the road lane markings of a road lane lyingwithin the field of view via pattern recognition and compute the highestallowable speed, in particular in curves, therefrom. If the informationsystem determines, independently or via connection to the instrumentsystem of a vehicle, that the vehicle has exceeded this computed highestspeed, then a warning symbol 1395 can be blended in in the region ofkeenest vision 1391. This is illustrated in FIG. 13B. The advantage ofblending in of a warning symbol 1395 in the region of keenest vision1391 lies in the fact that the symbol 1395 appears where the eye islooking and thus does not tempt the eye to look away from the presentscene. For this reason, the brightness of blended in symbols should bechosen such that the symbol appears translucent. Optionally, the drivingaid system can also acoustically warn a user about the danger.

The information system 100 may be a complex, multifaceted informationsystem. FIGS. 14A and 14B illustrate the use of the information system100 as a mobile fire department command center, according to anotherembodiment. The mobile fire department command center 1410 includes acommand console 1412 and several helmet systems 1411. Each of the helmetsystems 1411 comprises a signal capturing apparatus, as described above,and a field-of-view capturing apparatus. Each of the helmet systems 1411can optionally be equipped with a projection apparatus, infrared sensorsor position sensors. The helmet systems 1411 can also be equipped withfurther sensors that allow, e.g., an assessment of the air quality orwind speed. For the purpose of communication, each of the helmets 1411is equipped, for example, with a radio transmission unit thatcommunicates with the command center 1410 and/or the command console1412. The radio transmission unit can take over both tasks of aninformation unit as well as tasks of an output apparatus by transmittingand receiving information.

Preferably, the field-of-view images captured by the respective helmets1411 which can be brought into agreement with the truly perceived fieldof view of the respective firemen on the basis of the signals capturedfrom the eyes are transferred to the command console 1412 and presentedthere on monitors. In order to reduce the amount of data to betransmitted, users of the command console 1412 can also wear aprojecting spectacle system so that solely the image data falling ontothe region of the user's fovea centralis must be captured and/ortransmitted in high resolution. A correlated field-of-view image of anindividual fireman or a mosaic of several images could be projected intothe user's eye. Thus, the user could see exactly that which the firemansees or be provided an image from the fireman's field of vision thatchanges depending on his own eye movements.

In the optional case of a projection, additional information could bewoven into the image projected to the user and/or the fireman. Forexample, orientation and/or temperature information obtained via theposition sensors and/or infrared sensors could be blended into the fieldof view. Constantly blending in particular points on a compass such asNorth and West as well as altitude information could be helpfulreference information both to the user far away from the action he isseeing as well as to the fireman veiled in smoke and haze.

Through appropriate pre-processing of the captured position informationand due to the inherent networking of the system components, theposition of his colleagues, e.g. via a characterizing “X,” or theposition and severity of the sighted or otherwise captured hearts of thefire, e.g. via a dot that is colored in accordance with the strength ofthe fire, could be blended in to each fireman. This would make fightingthe fire easier and would reduce the probability of accidentallyinjuring a colleague hidden behind smoke or a wall.

FIG. 15 shows schematically an information system adapted for theoperation of an external system, e.g. a remote controlled robot 1579designed to move dangerous objects. A movable robot 1570 includes acamera apparatus 1571 as well as a grasp arm 1572. The robot 1570 isconnected to a spectacle system 1520 worn by a user 1502 e.g. via aradio connection. The images captured mono- or stereoscopically via thecamera apparatus 1571 can be mono/stereoscopically projected onto theretina of the user 1502 via a projection apparatus comprised by thespectacle system 1520. In the case of a stereoscopic projection, spatialvision would be ensured.

If the camera apparatus 1571 is provided with a macroscopic lens havinga wider “field of view” than the field of view of the user 1502, thenthe field of view seen by the user 1502 can be held in correlation withthe remote image as a function of the captured eye movements of the user1502 via a corresponding selection of an image detail from the imagesupplied by the camera apparatus 1571 as described above. The movementsof the head of the user 1502 can also be captured via position sensorssuch that the camera apparatus 1571 pivots in correlation with the headmovements. The information system in accordance with the invention thusoffers a previously unachieved degree of visual authenticity during theperception of a remote scene, which considerably eases the control ofsuch an external system 1570.

By attaching a microphone to the external system, in particular adirectional microphone that is directed as a function of the headposition or the gaze direction, in connection with the headphonearrangement on the spectacle system allows a further sensory dimensionto be realized.

In order to allow further operational control of the robot 1570, amanually operable joystick 1525 is connected to the spectacle system1520 e.g. via a cable 1526. Thus would allow, for instance, the grasparm 1572 or the direction of motion of the robot 1570 to be controlledin several directions.

FIG. 16 schematically illustrates another embodiment of the informationsystem using a spectacle system 1620, which acts as a universal remotecontrol for one or more devices, for instance a computer, a videorecorder 1676, a printer 1677, a slide projector and/or a telephone1679. The spectacle system 1620 provides an interface that communicatesin two directions between a user 1602 and any device 1675-1679 to becontrolled. First, the device 1675 through 1679 must be recognized. Thisis fundamentally carried out, by gazing at the device (1675-1679) to beoperated with the fovea centralis. The identity of the gazed-at device(1675-1679) can be determined either with or without the assistance ofthe device (1675-1679). In the following, it is assumed that both thedevice (1675-1679) as well as the spectacles 1620 are equipped with thesignal reception and/or transmission apparatus necessary for theoperations described.

If the identity is determined with the aid of the device 1675-1679, thenthis device 1675-1679 either radiates an ID-signal e.g. an infrared orultrasonic signal, in more or less regular intervals or it is requestedby a request signal radiated by the spectacles 1620 to radiate anID-signal. The request signal must be radiated localized to the gazedirection in order to avoid addressing other devices. The ID-signalradiated by the device 1675-1679 is recognized by the spectacles, as aresult of which conclusions are made re the identity of the device.

If the identity is determined without the aid of the device 1675-1697,then the spectacles 1620 carry out a pattern recognition of the gazed-atregion of the field of view in cooperation with a databank or otherinformation source 1640 that contains pattern recognition data for therespectively addressable devices 1675-1679.

Based on the identity of the device 1675-1679, a menu that is adapted tothe possible functions of the device is blended in, at a fixed location,into the field of view of the user 1602, if necessary upon the pressingof a button or the blinking of an eye. If the functionality of thespectacles is not readily known, then the corresponding information isfirst established from a databank or other information source 1640, e.g.via standardized interrogation of the device itself. In this case,identification of the device embedded into the interrogation signalensures that solely the desired device responds to the interrogation. Byblending in the menu into the field of view at a fixed location, theuser 1602 can control the menu, which may be hierarchical if necessary,via slight eye movements like a computer menu.

After the desired function has been selected, a signal corresponding tothe function is sent from the spectacles 1620 to the device 1675-1679.In this case, identification of the device embedded into the signal canensure that solely the desired device reacts to the signal. In thismanner, quick and easy operation of many devices can be achieved withlittle hardware.

FIG. 17 shows schematically an optical device with a hinged mirror 1755allows a switching between a capturing from the field of view and acapturing from the eye 1780 or a projection onto the retina 1781. Theadvantage of this optical device lies in that the same swiveling mirrors1754H and 1754V can be used for a capturing from the field of view andfor a projection onto the retina 1781 and that the beam path for acapturing from the field of view and the beam path for a capturing fromthe eye 1780/a projection onto the retina 1781 is, to a large degree,accordingly identical. In this manner, a high correlation between thelight captured from the field of view and the signals captured from theeye/a high correlation between the light captured from the field of viewand the image projected onto the retina is achieved through the opticalsystem itself. This means that no additional correlation errors arecaused by the aforementioned beam paths traveling across differentswiveling mirrors that could exhibit different rotation characteristics.For capturing light from the field of view and capturing light from theeye, even the same light capturing apparatus 1751 can be used. Thecorrelation can solely be negatively influenced by the reflection on thespectacle lens 1724 and the optical system of the eye 1780.

There are other embodiments directed to the use instead of a traditionalTV devices, newspapers, books, as helmets, diagnostic or treatmentdevices and other applications.

Previous electronic books and/or newspapers have the disadvantage ofbeing too heavy and/or too unwieldy and can moreover only present alimited amount of information per page. Portable video and TV devicesare also heavy and/or unwieldy. If the information system in accordancewith the invention is designed such that the provision of informationcomprises a projection of image information into the eye, then variousvision-related media, e.g. electronic books or newspapers, television orvideo games, can be implemented via the information system. In suchcase, the information system in accordance with the invention isimplemented, for example, in the form of wearable spectacles asdescribed above that can be connected e.g. to an information network, aportable storage apparatus, e.g. a CD-ROM or DVD-reading device oranother information source via a cable, infrared or radio connection.

An advantage of such a design of the information system in accordancewith the invention is that its capturing of signals from the eye inconjunction with a field-of-view capturing allows a projection in whichthe projected text and/or the projected images appear to be fixed inspace. For this purpose, the information apparatus comprises anevaluation apparatus that determines the correlation of the visual axisto the field of vision and that accordingly controls the projection suchthat the information projected onto the eye appears to be immovablevis-á-vis the field of vision in spite of movements of the eye. Thedetermining of the correlation of the visual axis to the surroundingscan also be assisted by position sensors mounted in the spectacles.

The virtual position of the fixation fixed e.g. via a fixation with theeye in conjunction with a blinking of the eyes or a pressing of a buttonor even automatically, for example, by using an image-processingevaluation of the field of vision that determines an area of the fieldof vision having as little content as possible. The disruptive effect ofthe natural field of view not necessarily covered up by the projectionof information can be reduced via a complementary-color “wiping out” inwhich complementary-colored pixels are determined on the basis of thelight captured from the field of view whose correlated projection ontothe respectively associated regions of the retina make the naturalbackground appear white on account of the addition of colors. If a blackbackground is desired, then the perceived total brightness of theprojection must exceed the perceived total brightness of the naturalfield of view by roughly 10% to 20% as described above so that even thebrightest points of the natural field of view are perceived as black.

For the sake of controlling operation, image information representingvirtual control knobs can be projected into the eye such that theylikewise appear fixed in the vicinity of the text and/or image in thefield of view. The virtual information medium could thus be remotecontrolled, i.e. page turning, fast forwarding, rewinding or the like,by gazing at the corresponding control knob with the fovea centralisplus pressing a button or blinking an eye. Similarly, access to lexical,databanks, etc. could be made possible by gazing at presented words orimage sections. Instead of control knobs, the information system couldalso be controlled, for example, via menu guidance in which controlmenus “pop up” when a particular region of the image is observed inorder to allow an ocularly controlled selection from the menu which maybe hierarchically constructed, if necessary.

A further advantage of such a design of the information system inaccordance with the invention is that the amount of data necessary for asufficient, momentary presentation is far less than the amount of datathat would be necessary for a high-resolution presentation of the entirefield of view. This is due to the fact that the information system hasknowledge of the region of keenest sight. Thus, only those portions ofthe projection must be carried out at high resolution that regard theregion of the fovea centralis. Onto the other regions of the retina, aprojection having a lower pixel density suffices. The amount of datanecessary for an instantaneous presentation is accordingly reduced,which has clear system advantages. In particular, the perceived size ofthe projected image may be arbitrarily chosen without unprocessablylarge amounts of data for presentation of the instantaneous image beingthe result.

If the projected image is larger than the field of view, then thecurrent visual axis determines the cropping of the image. The projectionis carried out such that the current image detail fills the entireactive region of the retina. By moving the eyes, further sections of theimage can be brought into the field of view. If the projected image issmaller than the field of view, then projection must only be carried outonto a limited portion of the retina. If the natural background of thefield of view is not blended out, then this changes during movements ofthe eyes. In particular for television or cinema-like presentations ofinformation, a projection that fills the field of view exactly ispreferred.

If signals are captured from both eyes of a user, then the projectioncan be carried out stereoscopically, wherein a slightly different imageis supplied to each eye such that the brain believes to perceives athree dimensional total image. This allows an optimal system-humaninterface e.g. for 3D television, 3D video games, 3D CAD applications orother, in particular interactive, 3D applications to be realized.Preferably, the information system comprises further control elements,for example a joy stick, pedal or steering wheel that allows anavigation and/or change of perspective within the presented virtualimage or other influencing of the presentation of information or of asystem connected with the information system. As described above, theeye itself can also act as a control element.

By accordingly applying the above measures necessary for the positioningof an electronic newspaper at a virtual location, it is likewisepossible to project the person wearing the information system inaccordance with the invention other orientation aids onto the retinasuch as, for example, an artificial horizon.

The information system 100 may be arranged for ophthalmologicalapplications and visual aids. Due to its capturing of signals reflectedback from the eye, the information system in accordance with theinvention is excellently suited for embodiment as an ophthalmologicalsystem. For example, the information system in accordance with theinvention can be implemented as a positioning system forophthalmological surgery, in particular for ophthalmological lasersurgery. The information system can also be used e.g. as anophthalmological diagnostic system, visual aid system and/or visualdeficiency correction system.

Most of the structures or organs of the eye are very small in comparisonto manual movements. Diseases and injuries to these structures/organsoften only affect a small, microscopic area. As opposed to many otherparts of the body, the eyes, however, cannot be fixed, which makes thetreatment of possible diseases or injuries to the eye particularlydifficult.

Due to the ability of the information system to exact follow movementsof the eye and provide information with regard to the momentary positionof the eye even to other systems, these difficulties can be overcome viaa therapeutic system on the basis of the information system. Forexample, the therapy system can be connected to the information systemfor the purpose of exchanging information in such a manner that theinformation with regard to the momentary position of the eye is providedto the therapy system such that a high-precision, automated therapy ofthe eye can be carried out even when the eye is moving.

In accordance with another embodiment, a therapeutic laser beam isdirected via the optical system. A laser treatment of the eye, inparticular of the retina, can thus be carried out in the same manner asa projection as described above. For example, diseased veins in thechoroid coat can be stultified in that a photosensitive preparation isinjected or taken in and that the diseased portions of the choroid coatare precisely irradiated for several tens of seconds. Such a therapy canbe precisely carried out with the aid of the information system.

In order to be used as a visual aid and/or visual deficiency correctionsystem, the output apparatus of the information system comprises aprojection apparatus that projects the vision-improving imageinformation onto the retina. In addition, the information apparatuscomprises an evaluation apparatus that determines the vision-improvingimage information on the basis of the light captured from the field ofview. The vision-improving image information is preferably projectedonto the retina in correlation with the signals captured from the eyesuch that the naturally perceived field of view and the projected imageinformation are perceived as a unitary image. In extreme cases, thevision-improving image information is projected onto the retina suchthat the otherwise naturally perceived field of view is not at allperceived by the eye. As described above, the degree of perception of animage projected in this manner in relation to the naturally perceivedimage can be controlled via the brightness of the projected pixels.

Such an information system allows e.g. vision deficiency correction forshort or far-sightedness as well as for color blindness to be carriedout. During the correction of short or far-sightedness, the informationsystem can set to a (quasi-)fixed correction, can allow a variablecorrection, or can automatically, dynamically adjust itself to thevisual deficiency. The correction is carried out via an adjustable (ifneed be), optical focusing system within the projection apparatus or viaimage processing measures. Latter can be implemented at low system cost.

Implementations with (quasi-)fixed or variable correction areunderstandable to the person skilled in the art without furtherexplanation due to inherent similarity to similar optical systems. Animplementation with a dynamic, automatic correction of the naturalimaging error comprises, in addition to the aforementioned correlation,a further dependency on the signals captured from the eye. In such case,in particular a retinal reflex image is captured that supplies, viacomparison with light captured from the field of view and/or via imageprocessing evaluation, information re the sharpness of the image imagedonto the retina. The light captured from the field of view isaccordingly processed into vision improving image information andprojected onto the retina. The information system can also act as adiagnostic system through output of the correction values determined inthis manner.

Due to its capturing of signals reflected back from the eye and lightoriginating from the field of view, the information system in accordancewith the invention is in a position, by using a correspondinglyprogrammed evaluation apparatus, to supply information about manyophthalmologically relevant characteristics of the eye. For example,squint angle, primary positions (PP), visual field testing even withcolors, threshold tests, standardized testing methods for glaucomadiagnosis, retinal function tests (e.g. ERG and VEP) even at selectedlocations and tests of the receptive fields can be carriedout/determined. The person skilled in the art selects the signals to becaptured from the eye for this purpose, the field-of-view stimulinecessary for this purpose and the processing algorithms necessary forthis purpose on the basis of his specialized knowledge, accordinglytaking into consideration the invention described above.

Whereas e.g. the keenness of the vision can be determined through anevaluation of signals reflected back from the eye and can besubsequently corrected, the correction of many other visual deficienciespresumes a system-independent determination of the deficiency, forexample by an ophthalmologist. A befitting setting of the correctioncarried out by the information system can be carried recursively orsimply.

In a recursive adjustment process, a correction is carried out by theinformation system in accordance with a previous setting while thevisual acuity of the person with defective vision is being tested. Onthe basis of the results of the tests, a new setting of the informationsystem is chosen. This process is repeatedly carried out until thevisual deficiency has been sufficiently compensated. In this manner, theinformation system acts equally as a diagnostic system since the visualdeficiency can be determined based on the best-correcting final setting.

In a simple setting processing, the visual acuity of the person withdefective vision is tested without any type of compensation. Based onthe results of the tests, a suitable setting of the information systemis chosen that then, in later operation, prepares the light capturedfrom the field of view into vision-improving image information inaccordance with this setting and projects it onto the retina. During thepreparation, e.g. particular spectral components or particular regionsof the field of view are emphasized or modified through other imageprocessing measures in accordance with the setting, i.e. the originalvisual deficiency.

For people suffering from night blindness, a visual aid can be realizedvia the information system in accordance with the invention, forexample, in that the light captured from the field of view, e.g. viahighly light-sensitive photo-detectors, is strongly amplified andprojected onto the retina. In this manner, the cones can be stimulatedin such manner that predominantly color, photopic vision instead ofscotopic vision takes place. Also, the maximally allowable brightness ofthe individually projected pixels is limited to a predeterminedthreshold value in order to avoid a glaring through brightlyilluminating objects such as street lamps and oncoming cars. Such asystem is thus also suitable as an anti-glare system since, if thebrightness of the entire field of view is raised, whereas the“excessive” brightness of individual pixels is left unchanged, then the“excessively” bright pixels are not perceived as being “excessively”bright. If the information apparatus also comprises an infrared sensorthat captures infrared light from the field of view, then additional,monochrome image information with regard to the field of view can beobtained by night or fog that can be transformed into the visiblespectral range in order to improve the image information alreadyobtained via the field-of-view capturing apparatus and the evaluationapparatus.

In general, the information system in accordance with the invention canalso be suitable for improving visual acuity. E.g. in the case of strongor weak contrasts or in the case of low brightness in the field of view,image information that is adjusted with regard to its brightness can beprojected into the eye in order to allow improved visual acuity.

The information system 100 may be integrated into a helmet such as afireman's helmet described above. Similar embodiments, e.g. as asoldier's, driver's, crane operator's, sportsman's or pilot's helmet orspectacles are possible. A soldier's helmet/spectacles on the basis ofthe information system in accordance with the invention could be of aidto the soldier, for example, for orientation and/or for targeting. Insuch case, the information apparatus of the information systempreferably comprises sensors and/or radio receivers that allow anextrasensory perception of the surroundings and/or the reception ofinformation from a command center. The output apparatus will provideinformation preferably visually, acoustically or tactually, e.g. in theform of short electric stimulating currents on the skin. Latter could beused to directly inform a soldier of the direction of a foreign objectapproaching from behind.

As a night vision device, the information system would also captureinfrared light from the field of view in addition to the capture ofvisible light from the field of view. As described above, imageinformation can be obtained from such captured infrared light and beemployed in the enhancement of image information to be projected intothe eye.

If the information apparatus comprises e.g. a GPS receiver, then thehelmet could project position information or orientation aids onto theretina. Preferably, the projection of such information into the eye iscarried out similar to the projection of an electronic newspaper. Thisavoids a distraction of the soldier since the information appears to befixed in space or vis-á-vis a neutral position of the eye. An adaptationof the image information to the background perceived therebehind for thesake of best possible readability also takes place via an evaluationapparatus belonging to the information apparatus.

Although a radio transmission or other data transmission from thesoldier to a command center is generally to be avoided from strategicreasons of camouflage, a transmission of field-of-view data correlatedto the eye movements of the soldier to a command center could also bemeaningful in particular cases.

In an embodiment that is particular interesting for soldiers, theinformation apparatus comprises one or more cameras that capture imagesfrom outside the field of view. The image information obtained in thismanner is then projected onto the retina via a projection apparatus. Thesupplementary image projected onto the field-of-view image could beprojected, for example, as an image within an image as a small image inthe corner of the natural or projected field-of-view image or appear asa longitudinal strip along the bottom edge. In this case, the capture ofsignals from the eye serves, together with the capture of the field ofview, to maintain the projected images in correlation with the movementsof the eye.

For a crane operator, it is helpful to project in supplementary imagesfrom other perspectives into the field of view. The information systemincludes supplementary sensors providing aid distance or weightinformation that is projected into the field of view. Such informationcan also be provided audibly or visually, e.g., upon gazing at the loadin combination with the clicking of a button. In this case, the lightdetermined from the field of view serves as a basis for the imagerecognition, whereas the signals from the eye allow a correlation of thecaptured field of view to the visual axis as described above.

The information system 100 can provide a pilot with many various typesof information. Via a connection to the information system of anairplane, relevant data such as flight altitude, speed or direction offlight or even an artificial horizon could be blended in to the pilot'sfield of view, for example, as described above. During landing, landingaid information could also be blended in that depict a virtual landingcorridor or indicate altitude or direction correction values. Inmilitary applications, friend/foe and targeting aid information could beprovided to the pilot. In this case, the gaze direction of the pilotplays a role both during the spatial blending in of the information aswell as during information selection. The pilot would like a flyingobject upon which he has fixed his eyes' gaze to be identified. If theidentification is carried out visually, he does not want the blending into cover any relevant areas of his field of view. In this case, dueconsideration must be given to the contrary requirements that therelevant regions of the field of view are typically imaged onto thefovea centralis but also that only those images that are projected ontothe fovea centralis are sharply imaged. Thus, an intelligent blending inmust be carried out in which the relevant regions of the field of vieware recognized, for example, via image recognition and not solely viathe orientation of the fovea centralis. In this respect, the informationsystem in accordance with the invention can also act as a sub-system tothe information system of the aircraft and provide information thereto.In this manner, e.g. information with regard to where the pilot islooking could be supplied to the aircraft information system by theinformation system in accordance with the invention and contribute thereto target capturing. In true action, the information system could locateenemy radar positions via sensors and depict their position togetherwith the associated landscape in three dimensions.

Various types of information could be provided to sportsmen via theinformation system in accordance with the invention as in the examplesabove. Orientation aids, speed information and/or enhanced field-of-viewinformation that allows better vision at dusk, at night, in rainy sprayor fog could be provided, for example, by projecting information intothe eye. A non-visual provision of the information is particularlysuitable in the case of low-content information. Similar to the aboveexamples, an information system in accordance with the invention worn bya sportsman could act as a sub-system of a sporting device or of avehicle.

The information system 100 can provide extrasensory perception using oneor more sensors, e.g. magnetic field detectors, pressure sensors,thermometers, spectral sensors, optical or acoustic interferencemeasuring devices. In particular in the case of a superimposition of apictorial presentation of information obtained from the sensors onto thenatural field of view via projection into the eye, the presentationcorresponds to the needs of the person having vision. In such a case,the information system in accordance with the invention can appear as acomponent, in particular as a presentation apparatus, of a complexmeasuring apparatus.

An example of such a system are spectacles equipped with sensitivemagnetic sensors that is in a position to locate current-conducting ormetallic objects in correlation to the spectacles. If such locatedobjects are designated, true to their position and in color, in thenatural field of view by projection that enables locating, e.g., waterpipes or electric wiring running under plaster. A handyman wearing sucha spectacle system would see the path of the piping or wiring as“painted on the wall”.

If a two or three dimensional array or other one or multi-dimensionaldistribution of the sensors is chosen, then even e.g. highly complexvector fields or gradients could be made visible to an observer as animage over the object or arrangement associated therewith. For example,an arrangement of pressure sensors around a test object in a wind tunnelcould supply pressure information that is prepared, in such a manner,via the information system in accordance with the invention as describedabove and projected into the eyes of an observer who is observing thetest object through a window such that he sees the pressure gradientsresulting from the test object there where they are present based onappropriate, colored depiction of the pressure values. Temperatureinformation obtained using an infrared camera could be presented to awelder in his field of view such that the local surface temperaturealong the work piece is recognizable.

Similarly, spectral sensors could be used to give a user informationabout exact color values or material compositions. In this case, it isalso practical to present the determined information audibly dependingon exactly where the user is looking. In conjunction with a databank andpattern recognition, such a system could be used, for example, to atleast approximately identify mushrooms or plants, wherein the user, uponsystem request, looks at particular parts of the mushroom/plant and/orturns these to face the sensors.

Having described various embodiments and implementations of the presentinvention, it should be apparent to those skilled in the relevant artthat the foregoing is illustrative only and not limiting, having beenpresented by way of example only. There are other embodiments orelements suitable for the above-described embodiments, described in theabove-listed publications, all of which are incorporated by reference asif fully reproduced herein. The functions of any one element may becarried out in various ways in alternative embodiments. Also, thefunctions of several elements may, in alternative embodiments, becarried out by fewer, or a single, element.

1. An information system for guiding a driver of a vehicle, comprising: a camera constructed and arranged to capture light from the field of view of said driver without capturing a retinal reflex image; an information unit, comprising a position sensor and configured and adapted to determine current positioning information of the vehicle; to provide additional data associated to the current positioning information; to extract information from said captured field-of-view light; and to determine a correlation between the extracted information and the additional data; and an output unit constructed and arranged to provide guidance information to said driver using said determined correlation provided by said information unit.
 2. The information system of claim 1, wherein the output unit comprises a projection unit constructed and arranged to receive an image signal representative of an image from the information unit and for visibly projecting said image to said driver.
 3. The information system of claim 2, further comprising an optical signal unit configured and adapted to capture optical signals reflected back from an eye of said driver; and the information unit is further determining a position and/or orientation of said eye based on said captured optical signals.
 4. The information system of claim 3, wherein the projection unit is configured to blend the guidance information into the field of view of said driver based on the determined position and/or orientation of said eye of said driver.
 5. The information system of claim 4, wherein the information unit further determines a suitable blend-in position from the captured field-of view light by means of an image-processing evaluation.
 6. The information system of claim 5, wherein a suitable blend-in position has little image texture.
 7. The information system of claim 6, wherein the projection is adapted based on the extracted information such that the natural background appears white or black.
 8. The information system of claim 1, wherein the information unit is configured to identify at least one landmark in the direct surroundings of said driver and the output unit provides information with respect to the identified landmark.
 9. The information system of claim 8, wherein the output unit only provides the information if the driver selects a physical or virtual activation button.
 10. The information system of claim 1, wherein the output unit provides acoustic guidance information.
 11. The information system of claim 1, wherein the guidance information is indicative of at least one of navigational information and driving aid information.
 12. The information system of claim 1, further comprising an acceleration measurement apparatus.
 13. The information system of claim 1, wherein the information unit is further constructed and arranged to determine the road lane markings of a road lane lying within the captured field of view; to compute the highest allowable speed with respect to that determined road; and the output unit issues a visual and/or acoustic warning to said driver when it is determined that the vehicle is exceeding the computed highest speed.
 14. An information system for aiding a driver of a vehicle, comprising: an optical signal unit configured and adapted to capture light reflected back from an eye of said driver; an information unit configured and adapted to determine a position and/or orientation of said eye based on said captured light; a distance sensor that determines a distance between said vehicle and at least one object located in front of said vehicle in a field of view of said driver; an evaluation apparatus that determines whether said vehicle is on a collision course with said at least one object; and a projection system configured and adapted to display, if said evaluation apparatus determines a collision course with said at least one object, a warning symbol into the field of view of said driver.
 15. The information system of claim 14, wherein the distance sensors is an optical signal unit configured for stereoscopic capturing of light from the field of view of said driver without capturing retinal reflex image.
 16. The information system of claim 14, further comprising: an optical signal capturing system configured and adapted to capture optical signals from said field of view without capturing a retinal reflex image; wherein, said projection system is configured and adapted to alter a brightness of said displayed warning symbol based on a brightness of said captured optical signals.
 17. The information system of claim 16, wherein said projection system is configured and adapted to alter a brightness of said displayed warning symbol based on a brightness of said captured optical signals such that said warning symbol appears translucent. 